During the initial stage of the project I tested gesture following and recognition, with a Microsoft Kinect camera and the Gesture Follower library for Max developed at Ircam. The physical adaptation of this technology to the project was hard to achieve, as the Kinect's range is limited relative to the stage size. The alternative was to use a single camera to capture the whole stage. This was a simple, efficient and reliable solution, which did not obstruct the dancers or limit the area where their movement could be detected.
Motion of the dancers is tracked with computer vision using a web camera connected to a laptop placed in the rig. The video is processed on the same laptop, because it is more efficient to send control data over the network than video captured at 120 frames per second. Within Max (Jitter) the stage is split into 20 sections. The amount of movement in each area is continuously calculated and all data is passed over the network using the UDP protocol to the other 3 computers at the control desk.
The sound engine is made with Pure Data, software by Miller Puckette. During performance the sound engine ran on two machines simultaneously. This was a backup solution in case one of the computers encountered any issues. The piece would continue to run at the cost of just a few seconds of interruption, the reaction time needed to switch the corresponding mixer channels on.
All the sounds are triggered live by dancers based on their location on stage. The volume of the sounds is also dynamically controlled based on the intensity of their movement.
Data received from all stage areas is compared. The 4 highest values, corresponding to the most active 4 places on the stage, are calculated over a varying time window, and then affect the volume of all sample players asynchronous. The time window can be changed during performance by the operator of the system. This manual control over the degree of intensity, enables varying the amount by which dancers affect the volume of the sounds. The sensing system reads whole body movement, such as changing position on the stage, as a greater amount of reduction than the “compression” gesture, but for the audience that shouldn’t appear as much a powerful change. This adaptation helps the audience to associate the “compression” gesture made by the dancers with keeping the lid on the sound, and the dancers to feel when they have control over the sound.
The pace of the “compression” gesture on screen is also determined by the movement of the dancers. This originated from the idea of visually representing the increased intensity in sound with a rapid change of images, in contrast with the immobile dancers. Therefore the video playback speed is determined by the same data that affects the sound. Video runs in Max on the graphics card, using existing openGL objects.
In the final section of the piece the image is split into 4 slices, which is meant to express a visual disintegration of the image, and show a clear relation between dancer's movements and the intensity with which the slices appear on screen, to emphasise the interactive side of the project.
The composition is a made of elements meant to work together under the given circumstances, to support the concept of the piece. The sounds are mechanical, industrial, hints of melody are rare. The sounds are composed in such a way that sudden changes in intensity are perceived as natural.
The system's design enables sounds to fade in and out at any moment in time, suddenly increase or decrease in amplitude, without this being perceived as an interruption.
All sonic elements have been pre-composed, in such a way that they work together in any order and number. All the sounds were made from scratch, no samples or pre-recorded materials were used, only extensive processing and editing.
Listen sound examples.