Integrating augmented reality to enhance expression, interaction & collaboration in live performances: A ballet dance case study

The democratization of high-end, affordable and off-the-shelf sensors and displays triggered an explosion in the exploration of interaction and projection in arts. Although mostly witnessed in interactive artistic installations (e.g. museums and exhibitions), performing arts also explore such technologies, using interaction and augmented reality as part of the performance. Such works often emerge from collaborations between artists and scientists. Despite being antonymic in appearance, we advocate that both fields can greatly benefit from this type of collaboration. Since 2006 the authors of this paper (from a research laboratory and a national ballet company) have collaborated on augmenting a ballet performance using a dancer´s movements for interaction. We focus on large productions using high-end motion capture and projection systems to allow dancers to interact with virtual elements on an augmented stage in front of several hundred people. To achieve this, we introduce an `augmented reality engineer´, whose role is to design the augmented reality systems and interactions according to a show´s aesthetic and choreographic message, and to control them during the performance alongside light and sound technicians. Our last production: Debussy3.0 is an augmented ballet based on La Mer by Claude Debussy, featuring body interactions by one of the dancers and backstage interactions by the augmented reality engineer. For the first time, we explored 3D stereoscopy as a display technique for augmented reality and interaction in realtime on stage. The show was presented at Biarritz Casino in December 2013 in front of around 700 people. In this paper, we present the Debussy3.0 augmented ballet both as a result of the use of augmented reality in performing arts and as a guiding thread to provide feedback on arts-science collaboration. First, we will describe how the ballet was constructed aesthetically, technically and in its choreography. We will discuss and provide feedb- ck on the use of motion capture and stereoscopy techniques in a live show and will then broaden the scope of discussion, providing feedback on art-science collaboration, the traps and benefits for both parties, and the positive repercussions it can bring to a laboratory when working on industrial projects.