THE CHOREOGRAPHIC LANGUAGE AGENT

Scott deLahunta (San Francisco, February 2009).

This draft was written for publication in the Proceedings from the World Dance Alliance Conference, 14-18 July 2008, Brisbane. Anticipated publication date: Autumn 2009.

Abstract:

Initiated by London-based choreographer Wayne McGregor and arts researcher Scott deLahunta in early 2000, ENTITY is part of an ongoing interdisciplinary research project aiming to broaden understanding of the unique blend of physical and mental processes that constitute dance and dance making. One of the research objectives is to apply this understanding to the design of software programmes that can augment the choreographer's creative process. The first of these programmes, the Choreographic Language Agent (CLA), is being built to generate unique solutions to choreographic problems, offering McGregor an alternative set of movement decisions to consider in the creation process. The CLA carries on the tradition of other contemporary choreographers, e.g. Merce Cunningham, Trisha Brown and William Forsythe, in exploring the potential of formal procedures for generating unique movement material through their dancers' interpretations. This essay discusses and contextualises the design work on the early CLA prototypes.

Essay

The paper notebook with its companion pencil or pen is a creative tool for many contemporary choreographers and their dancers. Using the notebook affords a relationship with a set of external objects inscribed 'on the page' in the form of drawn sketches, notations and diagrams combined with text. (Blackwell, deLahunta, McGregor, & Warwicker, 2004) This relationship can be described in cognitive terms; e.g. where the page becomes a surrogate for working memory or as a way for seeing something new by modelling structures or processes. The notebook in this sense becomes a site for the encounter of cognition and creativity; providing a place for thinking generatively with external objects (sketches, notations, etc.), an idea this essay will revisit. (Blackwell et al., 2004, p. 70) This concept that the choreographer's notebook affords a relationship with objects on the page, underpins the current development of the Choreographic Language Agent (CLA), as an extended interactive digital notebook.

The CLA evolved out of a vision London-based choreographer Wayne McGregor had for a suite of creative software tools that would use Artificial Intelligence (AI) algorithms to generate unique solutions to choreographic problems and augment McGregor's creative decision-making processes in the studio. The first real step towards developing this suite was taken in 2004, when McGregor and arts researcher Scott deLahunta began delving into the field of AI within the frame of a proposal entitled ENTITY: the autonomous choreographic agent. (1) This came after a six-month collaborative interdisciplinary project that aimed to draw on information from the field of cognitive psychology to enhance understanding of the choreographic process. This project, titled Choreography and Cognition, invited five scientists to work for short periods in residence with the dance company and resulted in a variety of studies including movement control, phrase segmentation and the cognitive dimension of notations. (2)

This initial engagement with cognitive psychologists was valuable preparation for the more in-depth AI research to follow. AI, along with psychology, neuroscience, linguistics and philosophy, is considered part of the broader study of the brain and mind that the cognitive science field has come to represent. (3) In early AI research, computers and the brain were thought to be performing similar processes; and modelling brain-like mechanisms in the computer was seen as a way of studying intelligence. (4) Following from this, as the ENTITY project aimed to build 'autonomous choreographic agents' using AI algorithms, it also envisioned researching and developing these 'agents' as a means to broaden understanding of the blend of physical and mental processes that constitute dance and dance making. Exploring what cognitive psychology might have to say specifically about the thinking process involved in dance creation was taken as the first step in this research, and outcomes of this examination of choreographic thinking are described later in the essay. (5)

To go beyond a superficial understanding of AI required at least a brief exploration of its historical relation to the development of cognitive science, its successes and failures, and why the particular class of AI problems and solutions known as 'agent-based' appeared to offer a way forward for the ENTITY project. (6/7) This laid the groundwork for conversations with specialists working in the field of AI and related areas. (8) Some specific writers and artists doing related work were considered. AARON, for example, is a computer program developed by Harold Cohen to autonomously produce aesthetically accomplished drawings. Starting out as a painter in 1971, Cohen became interested in AI and the possibility of building a 'machine-based simulation of the cognitive processes underlying the human act of drawing' (Cohen, 2001). Cohen has written a succession of increasingly complex software programmes, each a new version of AARON. Cohen's project features in Margaret Boden's 1990 book The Creative Mind: Myths and Mechanisms, in which Boden, founder of the School of Cognitive and Computing Sciences, Sussex University, explores how human intuition and creativity might be better understood with 'the help of ideas from artificial intelligence' (2004, p. 15). Boden's useful book notwithstanding, the field of AI with its communities of practice largely motivated by non-art engineering goals and computer science research questions, proved to be a difficult one within which to find a tractable approach for building creative choreographic tools.

In the artist and artificial intelligence researcher Marc Downie, the project found a collaborator with the overall sensibility and specific know-how to start the design phase of the autonomous choreographic agent. Downie had developed award-winning digital artworks using the 'agent-based' approach from AI before completing his 2005 PhD thesis entitled 'Choreographing the Extended Agent' at MIT's Media Lab. (9) Downie's thesis provides the background for his work as a member of the digital arts OpenEnded Group where he has collaborated with choreographers Trisha Brown, Bill T. Jones and Merce Cunningham (Downie, 2005). He approaches making agent-based artworks as 'a software engineering problem', but one that requires solutions that work for the art domain. This applied approach also differentiates his process from areas of AI research focusing mainly on computer science problems and the development of new algorithms to solve them.

Downie's proposal for an agent-based aesthetics is one that applies the algorithms of AI in the context of artistic creation; and the requirements of this context have given rise to the development of FIELD, a practical innovation important for the ENTITY project. As a programmer inspired by the challenge of collaborating with other artists more quickly in real-time, Downie developed FIELD as an authoring system to allow new algorithms and control structures to be rapidly created and revised as necessary to support creative interchange. Using a custom graphical environment, FIELD allowed his 'agent-based approach to meet the realities of collaboration, rehearsal and improvisatory choreographic practice' (Downie, 2005, p. xii).

FIELD has recently been released in a Beta version as open source software and is the environment of choice for the development of the Choreographic Language Agent (CLA). Also working on the CLA is composer and programmer Nick Rothwell. Based in London, Rothwell has worked on other dance projects, exploring the computational relationship between software and choreographic structure. For example, for ChoreoGraph (2002) he worked in collaboration with Austrian dance maker Michael Klien to create software that re-arranges the order of a duet each time it was performed by dancers from Ballet Frankfurt (deLahunta, 2002). The third member of the current CLA project team is Luke Church, whose areas of research include the development of programming interfaces for non-programmers. (10) Church will assist in devising the interface that allows a choreographer, in the first instance Wayne McGregor, to work alone in his or her hotel room with the Choreographic Language Agent. One of the potential advantages of this is that the CLA, being unlike a paper notebook and pencil, will support the mental harnessing of certain, possibly new, structures and relationships that will have a generative impact in the rehearsal studio. (11)

The CLA is envisioned as a small software environment, developed in FIELD, for exploring variations in choreographic instruction. Choreographic instructions refer to the types of problems, tasks, games and/or scores the dancers may be asked to solve or apply in the creation of movement material. As a method for generating movement material the choreographer may use or select from, the giving of instructions or formal tasks is common practice for a range of contemporary choreographers including Merce Cunningham, William Forsythe and Trisha Brown. (12) For McGregor, instructions stimulate a certain interior landscape of thinking that brings 'intentionality' (artist's use of the term) to the dancers' performance of the material. Instructions also support the dancers in their exploration of novel non-habitual movement patterns. Instructions are often derived from a conceptual or thematic starting point, for example an image of the Vitruvian Man, and may feature geometric lines or planes drawn both inside and outside the body space. They may also include procedures to be performed with these geometries and conditional statements governing the time or place in which actions may occur. Some instructions are extremely formal, based on numbers and coordinates; others are more narrative involving different forms of imagery through metaphor or triggers for non-visual sensory experience (acoustic, smell, somatosensory, etc.).

Notebooks are an important part of this process, used by the choreographer to work out possible instructions and for the dancers to work out features of possible solutions and recordings for future recall. As mentioned earlier, the notebook affords a relationship with a set of objects 'on the page' in the form of drawn sketches, notations and diagrams combined with text. Notebooks are things that McGregor and his dancers think with. The CLA aims for a similar quality of situated-ness in the creation process, as an extended interactive digital notebook to support creative problem-solving. As such, the CLA 'posits a new form of dance notation; one that aids the choreographer in generating dance movements rather than in recording existing movements' (Downie, 2008). The primary salient feature of the autonomous CLA, if it can be successfully developed, is that it will take instructions, interpret them and render dynamic graphic imagery as a potential proxy for the dancer; another problem-solving movement generating 'body' that McGregor and/or his dancers can refer to in the studio.

'Rather than attempting to produce a general model of movement, choreography, and meaning, the CLA focuses on the individual and even idiosyncratic methods of a given language movement system' (Downie, 2008); in this case as recorded in sets of choreographic instructions used by McGregor in a recent creation process, which have to be converted into machine-readable language. (13) The graphic rendering model takes as its point of departure a minimalist point-line-plane vocabulary instead of a sophisticated, anatomically correct joint hierarchy; the idea being to rapidly sketch movement explorations at all levels in a referent space (limb, body and stage-space). Given a sentence written in the converted language now known to the tool, the CLA can interpret this sentence to produce a short animation of its body. McGregor can then perform 'pseudo-linguistic operations on the language level, thus generating sequences, superpositions, and modulations' (Downie, 2008). After studying and labelling sets of sentences and correspondences between them, McGregor can start to determine the conditions under which an agent (or multiple agents) can autonomously deploy this language to generate possible movements. These may function as novel visual imagery to be then redistributed instruction-like to the dancers to work with, or even to create a piece of computer choreography.

To recapitulate, the core idea is to create CLA movements by writing sentences in a formal language inspired by taxonomies of instructions that have been used in a previous choreographic process (see Figure 1). The agent then automatically translates that sentence into animation by manipulating points, lines and planes inside a 3-D graphic visual space. (14) At the smallest level, one has these individual sentences that can be scrubbed through (showing you the animation) and then altered by editing the text. Multiple overlapping sentences can then be combined on a single timeline to construct a more sophisticated visual score. As mentioned earlier, McGregor will learn to work with the CLA by labelling sets of sentences and correspondences between them, over time building a unique collection of inspiring sentence-movements. In the future, the CLA will be shared with other choreographers invited to create their own collections; as a general tool for individual usage it is the sentence construction and labelling that should constitute the artist's signature. The software will include the possibility to specify conditions such as time and place when these sentences may be activated, thus forming a truly autonomous agent that makes its own decisions according to the language rules that McGregor and/or another choreographer writes.

Figure 1: Prototype example of sentence construction. Credit: Marc Downie/ OpenEnded Group

The graphic rendered 'body' or dancer proxy mentioned earlier is made up of a number of points loosely corresponding to a four-limbed body with head, a related movement space surrounding the body (or kinesphere) and to room space. This configuration serves as a reference, which is unconstrained in any real physical world sense. This means that the agent's body is quickly and easily made abstract so as not to compel an observer's projection of a real body onto it; as would naturally be the case if it were to appear too much like an actual body. The objective is to make it possible for McGregor to find the 'right balance of surprise, expectation and controllability' in working with the CLA (M. Downie, personal communication, 23 February 2009). The aim is to generate some level of intuition so that the choreographer can sense, but not know beforehand, the solutions the CLA might generate. As mentioned earlier, choreographers and dancers use notebooks as things to think with. They also use bodies to think with, as individual dancers with imagination and physical training and/or skills and a variety of attributes of personality (e.g. presence and charisma) afford a range of possible choreographic decisions (see Figure 2). The CLA is envisioned to make its contribution to the choreographic process situated somewhere between these two different areas: between notebooks and bodies.

Figure 2: Wayne McGregor working with dancers Paolo Mangiola and Neil Fleming Brown. Photo credit: Adriene Hughes.

Various researchers in philosophy and cognitive science working with dance are interested in how making is thinking and in making things to think with, and the CLA is a contribution to this area. (15) Thinking returns us to cognitive psychology and an ongoing collaboration with Cognitive Psychologist Phil Barnard. A key participant in the earlier Choreography and Cognition project and now Special Advisor to Wayne McGregor|Random Dance, Barnard has been working with McGregor on a 'theory of choreographic thinking'. (16) This has led to the articulation of models aiming to capture and make visible aspects of McGregor's choreographic process. The aim for this is not to produce only theory, but to be able to stand in and function as a collection of Choreographic Thinking Tools that can assist choreographers through augmenting their creative capabilities (deLahunta, Barnard & McGregor, 2009). One of these models is a perspective on the full range of mental resources (thinking components; forms and flows of imagery) that may be used in choreographic thinking. (17) As an extended notebook that deals directly with the instructions that take advantage of these mental resources, the CLA will be an intriguing research resource for the development of the Choreographic Thinking Tools.

As a single ongoing research project spread over several years, McGregor's and his collaborators pursuit of autonomous choreographic agents has engendered many interdisciplinary exchanges which have been generative of new thinking in the field of choreography and beyond. A select number of those connections have been mentioned in this paper. The Choreographic Language Agent is scheduled to be ready for use in McGregor's next creation process with his own company in the autumn of 2010. The premiere of this new dance is scheduled that November, and Downie and the OpenEnded Group are making plans to build a stand-alone installation involving several agents to open at the same time. It is intended that following this the CLA will be shared with others; and a new phase of research into choreographic practice will begin.

Acknowledgements: I am indebted to Marc Downie who is frequently quoted and paraphrased in the description of the Choreographic Language Agent that appears in this paper (from this source).

Endnotes:

1. A first draft of this was written in May 2004 and circulated to advisors and collaborators.

2. For more details of these studies, see the documentation website for: Choreography and Cognition (2003-2004). http://www.choreocog.net (accessed 18.02.09)

3. For a recently published history of cognitive science including in-depth material on Artificial Intelligence see Margaret Boden's (2006) Mind as Machine: A History of Cognitive Science. Vols. 1 and 2, Oxford University Press. For a quick glance at a description of cognitive science including history and methods see: http://plato.stanford.edu/entries/cognitive-science/ (accessed 20.02.09)

4. Hence it is no surprise that one of the most influential books from that time was titled Design for a Brain by W. Ross Ashby first published in 1952.

5. The meaning of 'thinking' as used in this paper is inclusive of tacit non-propositional forms.

6. For more on the history of cognitive science: Dupey, Jean-Pierre. (2000) The Mechanization of the Mind. Princeton University Press.

7. An agent is seen as part of an overall intelligent system in which the agent can perceive its environment and take autonomous action based on these perceptions; real world examples include software agents that search the internet for goods and services and data mining agents. For more on agent-based artificial intelligence see endnote 9.

8. These conversations comprised key meetings with specialists in a range of fields including robotics, computer vision, artificial intelligence, computational neuroscience, etc. involving a visit to the University of San Diego, California in April 2007 followed by the ENTITY Think Tank in London in November 2007 where some of the UCSD scientists were invited to join several from the UK in an interdisciplinary 'brain storming' session to envision autonomous choreographic agents.

9. For more on agent-based artificial intelligence: Maes, Pattie (1994) Modeling Adaptive Autonomous Agents. MIT Media Laboratory.

10. Luke Church is a member of Professor Alan Blackwell's Graphics and Interactions Research Group, (http://www.cl.cam.ac.uk/research/rainbow/research/) Computer Laboratory, University of Cambridge.

11. The aim of the project is not one of replacing dancers or to produce something that can save costs of dancer and studio time. The aim is to augment or to enhance, but also potentially challenge, lengthen, disrupt or perturb the creation work while informing it.

12. For example: Cunningham's use of chance techniques dating back to the 1950s; Brown's algorithmically determined dance structures include Locus 1975; William Forsythe's Improvisation Technologies methods were published as a CD-ROM in 1999.

13. Research into the giving and interpretation of instructions and problem solving in connection to distributed cognition is the focus of a group of cognitive science researchers based at University of California San Diego led by Professor David Kirsh.

14. By the time this essay is published there should be a short documentary movie on the website of Wayne McGregor|Random Dance in the R-Research section (http://www.randomdance.org/r_research) which will show how the sketching movements of the CLA are intended to work.

15. For example, David Kirsh and Alva Noë are a cognitive scientist and philosopher both working on theories related to 'thinking with things' in connection to choreographic research projects with Wayne McGregor and William Forsythe. For further reading: Kirsh, David (2009). Problem Solving and Situated Cognition. In The Cambridge Handbook of Situated Cognition. Philip Robbins and Murat Aydede (Eds.). (pp. 264-305). New York: Cambridge University Press.

16. Phil Barnard is Cognitive Psychologist with the MRC's Cognition and Brain Studies Unit, Cambridge.

17. The Interacting Cognitive Subsytems Model was developed by Barnard and colleagues as a macro theory for cognition and emotion. To read more: http://www.mrc-cbu.cam.ac.uk/research/emotion/cemhp/phil.barnard/ (accessed 17.02.09)

References

Boden, M. (2004). The Creative Mind: Myths and Mechanisms. Second Edition. London: Routledge.

Blackwell, A., deLahunta, S., McGregor, W., & Warwicker, J. (2004). Transactables. Performance Research, On The Page, 9(2), 67-72.

Cohen, H. (2001). Biography of Harold Cohen creator of Aaron. Retrieved 3 May 2009 from http://www.kurzweilcyberart.com/aaron/hi_cohenbio.html.

deLahunta, S., Barnard, P., & McGregor, W. (upcoming 2009). Augmenting Choreography: using insights from Cognitive Science. In . J. Butterworth & L. Wildschut (Eds.), Routledge Reader in Contemporary Choreography. London: Routledge.

deLahunta, S. (2002). Duplex/ ChoreoGraph: in conversation with Barriedale Operahouse. 2 May. Retrieved 15 February 2009, from http://www.sdela.dds.nl/sfd/frankfin.html.

Downie, M. (2005). Choreographing the Extended Agent: performance graphics for dance theater (PhD Thesis). Boston: MIT. Retrieved 3 May 2009 from http://www.openendedgroup.com/index.php/publications/thesis-downie/.

Downie, M. (2008). Choreographic Language Agent. Retrieved 3 May 2009 from CLA at: http://www.openendedgroup.com/index.php/in-progress/choreographic-language-agent/.