Making of the Spatialisation Effect - Pure Data/ Max/ Controller

Pure data effect

I started from the idea of projecting light through holes in a piece of paper. By moving the paper, the size of the individual light spots would vary. Each individual light would then be tracked with motion detection via computer vision and software which is now freely available, after previously only being available in science labs, or at expensive development level.

The data obtained would in turn affect the delay times within a delay network, using power conservation. More details can be found in the chapter about delay networks in Miller Puckette's The Theory and Technique of Electronic Music.

Controller

The project developed from here into building a physical controller to affect the number of delays that interact with each other.

I liked the visual aesthetic of the warm light from the incandescent bulb of the overhead projector, and the sterile aspect of the grey scale image it creates. I was going for a visual reduction to the bare minimum needed for the initial idea to work, eliminating colour and fancy graphics to bring the image as close as possible in the area of light and shade, in a way I was trying to remove the object.

My plan was to build a device that sits on top of the projector, and affects the amount of light on the screen. The perforated paper idea proved unworkable because the image created was too fuzzy for reliable detection. Consequently, I have taken the idea in the opposite direction and experimented with a ball moving inside of a tube, thus inverting light with shade. Initial tests with motion tracking gave good results.

I liked in Bernier's frequencies (a) the design of the component pieces, the minimal use of materials, transparent plastic, silver metal of the tuning forks and triggering mechanism. I tried to get a similar look for the controller device which the audience can see during the performance. The clear perspex tubes, metal balls and pins, and grey foam provide that kind of simple look. The choice of wood instead of aluminium for the handles and base was just a constraint related to the need to build it myself and make it fit exactly on the projector. Bernier's black and white look of the room, the result of using strobe light, was the other aesthetic parallel with my project, in an attempt to simplify and suggest light and shade controlling the sound.

The dimensions and ideal number of tubes were determined on visual considerations, constrained by the size of the projector's bed. The number of balls within each tube had to satisfy both practical and visual requirements. They had to reach different degrees of speed in a short time. The tubes sit aligned parallel on the projector, on foam that acts as a spring. The balls are set in motion by depressing the ends of the tubes.

The motion tracking system requires the balls to move predictably and reliably. In order to move each tube independently, and to keep the movement vertical, the end of each tube is supported by an individual piece of foam. Adjacent blocks of foam are separated by a piece of thin plastic sheet, which allows the tube to spring back to the original position after being released.

Controller on screen from Andrei Branea on Vimeo.

Images with different amounts of colour correction.

After image reduction what is left on the screen are effectively moving dots, called blobs, corresponding to the balls and which are then individually tracked. This data comes in a block in which, at any moment in time, the data representing each blob is organised based on its vertical position on screen. If a blob changes size slightly, another blob on the same vertical line takes its place in the data output matrix. This causes confusion. Implementing a sorting algorithm from the same library was the solution to fix data getting mixed, and made tracking each blob continuously possible.

Motion detection capture after difference

Blob detection

Enhanced blob detection

Processing

The way in which the patch works is extremely simple. Data from each ball is always affecting one delay, in effect the delay time and gain is controlled by the location and speed of the balls. Because all delays change dynamically over time, the power conservation assures that the system doesn't go into digital distortion.

In order to avoid audible clicks when the delay time is changed I use an interpolation scheme. This means that each delay is actually made from several delays playing the same audio, which are faded in and out consecutively. I have matched the ideal camera frame rate, necessary for a smooth detection of the balls, with the interpolation scheme as closely as possible.

At 60 fps data from the camera is refreshed every approximately 16 ms. Using 8 delays within the interpolation scheme, with 128 volume fade points for each delay, and a metronome set at the minimum possible of 1 millisecond, the delay time can change every 16 ms without any clicks being heard. The data rate is closely matched between the camera, motion detection and delay changes. For efficiency and tight synchronisation all delays read their amplitude steps from the same Hanning window, and the total volume of the summed delays in each interpolation subpatch is always constant. This assures that only a minimum amount of harmonic distortion is introduced.