Virtual Dance

A report based on the Riverbed Residency in UCI Motion Capture Studio

By Scott deLahunta, May 2001

CONTEXT FOR A MOTION CAPTURE RESIDENCY

The Riverbed Residency took place from 9-20 April 2001 in the Motion Capture Studio in the Music and New Media Building at the University of California, Irvine, Claire Trevor School for the Arts. I arrived as an observer at the beginning of the second week. The artists-in-residence were Paul Kaiser and Shelley Eshkar of Riverbed Digital Arts (http://www.riverbed.com) and Michael Girard and Susan Amkraut of Unreal Pictures (http://www.unrealpictures.com). These artists are well known in the dance field for using digital motion capture and animation technologies to create "Hand Drawn Spaces" (1998) and "Biped" (1999) with Merce Cunningham and "Ghostcatching" (1999) with Bill T. Jones. This residency at UCI's Motion Capture Studio provided them the unique opportunity of working in a non-commercial environment to capture/ gather material for their upcoming new work, an installation titled "Pedestrian". In the UCI Motion Capture Studio, they worked without the pressure of time deadlines and the hard concrete floors of commercial capture facilities and with the readily available skills and passions of dance artists. In return, they opened up their creative process in the studio, gave classes and lecture demonstrations to faculty and students of the Dance, Studio Art and Music Departments and an open public lecture on April 18th in Winfred Smith Hall on the UCI campus.

The Residency was organised by Lisa Naugle, Assistant Professor in the Dance Department with support from a grant from the Chancellor's Distinguished Fellows Research Program. Professor Naugle has been with UCI's Dance Department for three years during which time she has been involved in the introduction of a range of new courses and collaborative projects featuring different technologies into the UCI Graduate student curriculum including Active Space/ Interactive Performance Systems and telematic projects of varying descriptions (see her website for details http://www.arts.uci.edu/lnaugle). In 1998, she began to conduct classes in motion capture at an off-site location as part of an Internship Program with a local motion capture house. One of her first students, Amy Cady, completed her graduate thesis project in 1999 focusing on motion capture technology and its ability to represent Rudolf Laban's Space Harmony theory (see http://www.arts.uci.edu/abcady/thesis.htm), and additional graduate research is underway, i.e. Irma Castillo's thesis in Choreography and Motion Capture.

In 1999, the UCI's Claire Trevor School for the Arts installed their own motion capture system (the optical Vicon 8 http://www.vicon.com) to be integrated into the evolving interdisciplinary arts and technology curriculum with a particular focus on use by the Dance Department (as far as I am aware the first motion capture system in the world to be dedicated to a university dance department). This evolving curriculum already contained digital arts courses given by Dance Department Professors Mary Corey, who has taught a graduate dance course in interactive multimedia design for CD-ROM since 1997, and Alan Terricciano, who offers a digital sound course for graduate dance students as well as web design classes. Adding to this base, Professor Naugle and Professor Christopher Dobrian of the Music Department have organised and are advising interdisciplinary projects involving dance, music and computing science and engineering students some of which are now involving the new motion capture system, e.g., Frédéric Bevilacqua, a composer and computer scientist working at UCI as a post-doc at the Beckman Laser Institute, is collaborating with Professor Naugle on mapping motion capture points from a human figure animation via MAX/MSP to a variety of musical parameters.

Currently, the School for the Arts is investigating the possibility of someone joining the faculty who will assist in integrating digital 3-D figure animation into the arts and technology curriculum - a process that has gained momentum as the result of the Riverbed Residency. The success of the Riverbed Residency has led immediately to the planning of future artists residencies as well as inter-institutional partnerships with other dance departments such as Ohio State University (recently equipped with a new motion capture studio), and the Dance Department is preparing for several new graduate students to focus specifically on motion capture and animation. With all of this activity, the UCI Motion Capture Studio seems poised to make a major contribution to the integration of choreographic and dance practice with developments in the area of human figure motion analysis, synthesis and animation.

A more detailed description of the UCI Motion Capture Studio and some of the current and future projects and plans including curriculum development, residencies and workshops, etc. are available on this site. For more information about the work of Riverbed Digital Arts and plans for the "Pedestrian" installation and other work, please visit their site: http://www.riverbed.com. Paul Kaiser maintains this website which features current projects, an archive of past work and various papers he has written including valuable transcripts of interviews with some of his collaborators, including Michael Girard and Susan Amkraut (see details about accessing this interview below). I strongly recommend this particular interview as essential reading for anyone who finds motion capture and figure animation interesting as it provides insight into their early ground breaking contributions to the field.

CAPTURING MOTION FOR PEDESTRIAN

"Pedestrian" is scheduled to premiere in the autumn of 2001 in New York City. It is intended to be a film of about 12 minutes in length projected in a public installation setting in the guise of a street lamp casting its light (in this case a projection) onto the sidewalk (for an illustration visit Riverbed site). What the viewer/ audience member will see on the sidewalk below them are small human figures moving about in crowd-like patterns with the occasional interjection of dramatic actions and scenes. These figures will be proportionately of a size one might expect if you were looking down from a height of several stories, as if looking directly down from a window high above an open plaza scene below. Their movement will be derived from motion capture data targeted to a variety of human character forms and shapes - male/ female, young/ old, with hats/ without, etc. - modelled in Character Studio (the state of the art extension of the industry standard 3-D animation program - 3-D Studio Max) and the individual pathways and large crowd patterns of movement will be digitally choreographed using a combination of the Motion Flow Editor and the Crowd toolsets which come with the most recent version #3 of Character Studio. I will explain the significance of these software tools a little later, returning now to the work done during the UCI Residency.

Using the eight camera Vicon8 system in UCI's Motion Capture Studio, the artists set about capturing the movements that will be used to animate the figures in "Pedestrian". Here is a short description of one of those sessions:

This article assumes some knowledge of motion capture technologies by the reader (there are references and links below to more basic information), but I will add here that the Vicon8 system is an optical motion capture system which means it uses an infrared camera based system that records the position of small grey reflectors (usually in the shape of a ball) positioned at strategic locations on the body. In the UCI Motion Capture Studio they are placing up to 32 markers on each dancer and capturing two people simultaneously (to give some perspective, when Riverbed worked with Merce Cunningham three years ago to make "Hand Drawn Spaces" they were capturing just 22 markers on only one person at a time - the technology has moved on).

In the end, I believe Riverbed captured between 300 and 400 very short, 5-30 second, clips of movement that will become part of the "Pedestrian" palette. My choice of the word palette here is deliberate because it is in the nature of the process of this work that the final form of "Pedestrian" will become manifest as the animation artists and their software select the most appropriate materials for the piece from the palette of all of the movement captured. This includes movement captured from themselves (Paul, Shelley and Michael), Lisa Naugle and UCI dance students. During the residency period, William Forsythe (director of the Ballett Frankfurt) visited the UCI Arts College to consider possible future work in the motion capture studio during which time Lisa Naugle and he were motion captured improvising together, but this data will not become part of the "Pedestrian" palette.

Now that the motion data (from the positions of these 32 markers) is recorded in the computer, it will go through the laborious process of 'cleaning'. This means removing anomalies/ inaccuracies that occurred during the capture and solving for 'occlusion' which is when a marker is hidden from a camera. Comments from the Riverbed artists suggest it will take someone (probably Michael Girard) up to two weeks full time to clean up the data. For the Riverbed artists, the next step after cleaning will be porting into the Biped tool (a fully articulate human figure pre-prepared for the motion capture data) in Character Studio 3 where they will move into the next stage of the process of creating "Pedestrian". This next phase will make particular use of the software designed and created by Michael Girard and Susan Amkraut.

EMERGENT / EMBODIED KNOWLEDGE

During one of their lecture demonstrations Shelley Eshkar and Michael Girard showed a visualisation of what "Pedestrian" will look like - projected on the wall. We watched three dancing figures as if from above slicing through the space in unison; circling around themselves using large loping loose movements. This movement had been motion captured, targeted onto an animated figure and 'cloned' three times. After a brief period, a crowd of jogger 'clones' (one motion captured jogger just copied many times) entered the space of the screen from above. Like a swarm they began to surround the dancers who kept dancing, somehow managing to just barely avoid colliding either with each other or with the dancers. Yet, there must have been a few dozen joggers by this point filling the space of the screen… remember this is as if seen from above so you can clearly see all the changes in direction. The joggers were obviously turning just at the point at which they were about to collide with something. They also appeared to always be moving towards the dancers whenever their paths were not obstructed by something else. The effect was of some sort of organic order bringing to mind images of animals schooling and swarming. The patterns that emerged appeared to be complex, but in fact the rules to this game were simple.

The way this visualisation was assembled was by fixing the dancers' pathways first using different parts of the Character Studio software such as the Motion Flow Editor which enables the animator to string different motion captured fragments together, and the computer software will smooth out the transitions between them. Interpolation is the word used for this process that is partly based on the software's ability to use physical modelling to generate movements that might not have been captured.

Once the dancer's pathways are fixed, another software tool that has only been released with the recent Character Studio 3 release called Crowd is used to apply the simple rules to the joggers that result in the complicated emergent patterns we see on the screen. In the software documentation, these rules come under the heading of 'advanced behavioural animation' and include such things as avoid and orientation. The rules applied to the joggers were to avoid other movers and continually orientate towards the area of the dancers. And this is how they intend to build the scenes for "Pedestrian". Shelley Eshkar says he will "work out the scripts for something small like a 'handbag being stolen' and then Michael will build the rules for the crowd to move in and around this event… and this way we will get a mix of dramatic with emergent behaviour".

Simple rules leading to complex results - emergent and unpredictable - is a conception underlying the computer aided study of chaos and other complex systems. Artificial Life refers to the modelling of such systems in the computer. Crowd, the new tool created by Michael Girard and Susan Amkraut for Character Studio, supports the organisation of the behaviour of large numbers of animated figures by drawing on the same concept of simple rules leading to complex results. Crowd in particular uses some of the principles of flocking. Susan Amkraut is credited with some of the early work in the mid to late 80 on flocking systems along with Craig Reynolds (see http://www.red3d.com/cwr/boids/). Flocking systems used three simple rules to simulate what ornithologists had thought was the result of each bird following some equally complex set of rules. The simple rules to flocking are: 1) Separation: steer to avoid crowding local flockmates; 2) Alignment: steer towards the average heading of local flockmates; 3) Cohesion: steer to move toward the average position of local flockmates.

SETTING UP FOR THE FUTURE

'Virtual Dance' is a conjunction I have avoided in the past for its vagueness unless perhaps it was being discussed in reference to Suzanne Langer's classic discussion of the aesthetics of dance in Feeling and Form (1953) in which she refers to dance as "virtual gesture". However, 'Virtual Dance' has gained new currency for me in meeting Michael Girard (Susan Amkraut had departed just before I arrived) during the UCI Riverbed Residency. He has effectively demonstrated to me the possibility that we may see a new form of dance making arising out of a combination of motion captured movement, physical modelling (using software to increase the resolution or fidelity of the captured movement as well as providing further manipulation and editing possibilities) and the application of new animation techniques that will use simple rules to govern the behaviour of large numbers of figures. The concept of 'new' here is defined less by the forms that we, as viewers or audiences, may see manifesting from this convergence, but more by the evolving set of creative thinking practices that will be applied in the creation of these forms.

This is 'Virtual Dance', making dances for the screen using computer animation techniques in which abstract structures of information condensed and compiled in the computer are brought to bear on the creative project of capturing and manipulating the nuances of physical expression. Virtual Dance shares certain compositional processes of dance making with computational processes of problem solving. It is occurring in the moment that the choreographer/ animator sitting at his or her screen applies a set of instructions that achieve a satisfying result. Michael Girard said that "choreographers think in motion flow networks", and this was part of what inspired he and Susan Amkraut to create the Motion Flow Editor during the period of working with Merce Cunningham on "Hand Drawn Spaces". But who sits down in front of the computer and actually applies the Motion Flow Editor, knows how to get it to work, which menus lead to which results, where to click and how to drag and drop fragments of motion captured material; and under what conditions and in which roles do they perform this creative work?

During Riverbed's final public presentation during their UCI Residency, a student asked the question, "How are you going to inspire choreographers to work this way?" Paul Kaiser's response was he didn't really have a direct answer to that question, but he likes to speculate that having the experience of working with motion capture will influence one's choreographic practice in making actual dances. This will undoubtedly be true to a certain degree as juxtapositions of different types of creative thinking and practice usually do give rise to a certain sharing of processes. But I wonder how this connection might be more explicitly exposed and nurtured, in particular within an educational environment such as that at UCI. Perhaps there was a small indication of this in the opportunity that arose for Michael Girard while I was observing the Residency. After one of his lecture-demonstrations, Lisa Naugle persuaded him to come to her dance improvisation class - to try out some of his ideas for behavioural rules for governing group movement.

I observed a part of this class - and it was fascinating to see Girard, someone who is able to translate the complicated physics of movement into the digital space of the computer, explore possibilities with bodies in real space. I hope he has further opportunities to work this way. The Riverbed team came to UCI primarily to capture movement - hopefully future residencies will also give the dancers the opportunity to see their captured movement rendered into forms swirling and swarming about the space of the screen in ways it would be impossible to conceive without the software created by Michael and Susan. It occurs to me that in order for the sharing and overlap of these processes to be enhanced, it will be important to establish the conditions of research that will allow computer animators and motion capture specialists, choreographers and dancers to work together in the same spaces for even longer periods of time, but perhaps without forcing the collaborations to take on too much, to allow for temporary sharing and contingencies and to recognise where different creative thinking processes diverge and converge.

21 May 2001

Scott deLahunta

(thanks to Lisa Naugle for assistance in editing this report)

END/ END/ END/ END/ END

Riverbed - for information about their other motion capture, animation and dance projects and in particular the interview Paul Kaiser did with Michael Girard and Susan Amkraut which can be found by clicking on the "Ideas" link.

For a useful glossary and bibliography related to terms such as Artificial Life, algorithm and complexity put together by Jeffrey Goldstein of the Plexus Institute. Another site with definitions of A-Life is here: http://lslwww.epfl.ch/~moshes/alife.html

John Lansdown wrote this extremely useful paper on "Computer Generated Choreography Revisited" to present at the 4-D Dynamics Conference in September 1995. "After briefly reviewing the ways in which computers have been used in choreography and more general body modelling, the author goes on to describe his own work on using computes to choreograph dances by rule based techniques." A must read (and you better save a copy before the link goes dead).

Where you can purchase/ license copies of Character Studio 3 and read about some of the features and functions of the various toolsets including Crowd, Motion Flow Editor and Biped.

Some of the terminology used in the above article may not be familiar to you such as 'physical modelling'. However, descriptions of most of the terms, especially if they relate to computation, can be found on the internet (easily). Here is a short description from the above site: "Physical modelling is a method of producing physically correct motion for bodies in computer generated worlds, such as a heavy object falling under the influence of gravity."

The Digital Theatre Experimentarium was an interdisciplinary project involving a wireless magnetic motion capture system that took place from February to June 1999 in Aarhus Denmark. This is the archived project site with some useful, although slightly dated, information.

An DRAFT online article entitled "Choreographing in Bits and Bytes" by Scott deLahunta that elaborates some on the history of motion capture and on some projects involving dance makers using motion capture technologies. The images are LARGE and will be slow to download over a dial up connection. There are some good references at the end of the document, although some of the links are dead - which only means you need to search the net for the articles, for example:

Wes Trager's useful and basic "A Practical Approach to Motion Capture: Acclaim's optical motion capture system" written and presented at SIGGRAPH in 1994

David Sturman's very useful "A Brief History of Motion Capture for Computer Character Animation" also presented at SIGGRAPH in 1994.

Scott deLahunta began in the arts as a dancer and choreographer. Since 1992, as a partner of Writing Research Associates (WRA), he has organised several international workshop/ symposia projects in the field of performance including recently the third session of Conversations on Choreography at the Institute for Choreography and Dance, Cork, Ireland. He taught theory and composition classes from 1994-1998 at the School for New Dance Development, Amsterdam and returns there to teach on occasion. In 1996, WRA organised Connecting Bodies, the first conference in the Netherlands on the theme of the overlap between dance and emerging technologies. From February-May 1999, Mr. deLahunta was a guest professor with the Department of Dramaturgy, Aarhus University, Denmark where he was also co-organiser of the Digital Theatre Experimentarium, a project investigating the relationship between motion capture, animation and live performance. From 1998 to 2000, he was a consultant for the Laban Centre London on dance and technology applications and implementation. He is frequently invited to give presentations and contribute to publications on the overlap between dance and new media technologies at both small and large scale events such as DEAF (Rotterdam) and ISEA2000 (Paris). He is currently a Research Fellow and guest lecturer at Dartington College of Arts, UK where he is conducting research, also supported by the Arts Council England, into the conditions for collaborations between performing arts and applied science practitioners. In Autumn 2001, the WRA initiative *Software for Dancers* will conduct the first in a series of research labs/ thinktanks looking to develop new software tools for performance artists. Several relevant weblinks can be found here: http://huizen.dds.nl/~sdela/main.html