Prof. Sha Xin Wei — PI
Prof. Marcelo Wanderley — PI
David Gauthier — mechatronics & feature extraction programming
Freida Abtan   — sound instrument creator (software)
David Birnbaum — sound instrument creator
Elliot Sinyor  — sound instrument creator
Harry Smoak — assistant project technical coordinator
Doug van Nort   — gesture/sound feature extraction, mapping
Rodolphe Koehly  — physical materials advisor 

As an extension of the research work conducted with the Topological Media Lab (TML), Sha Xin Wei and his team are creating textile objects such as wall hangings, blankets, scarves, and jewelry that create sound as they are approached or manipulated. These sonic blankets can be used for improvised play.  A phonetic pun on the old acronym for What You See is What You Get from the era of the Graphical User Interface, WYSIWYG (for wearable, sonic instrument, with gesture) draws on music technology, dance, children’s group games, textile arts, and fashion. Created first and foremost to sustain social play for people of all ages, WYSIWYG allows players to express themselves whether enjoying time in a park, dancing at a club, passing the time during a long car trip, or just playing at home. The research is being carried out in collaboration with Marcelo Wanderley, an associate professor at the McGill University Schulich School of Music in Montreal, and draws on Wanderley’s research into the gestural control of sound synthesis and new interfaces for musical expression.

The custom-designed digital instruments embedded in the cloth sample movement to transform ambient body movement and freehand gestures into new sounds or “voices” associated with a player or transmitted to other players in the vicinity. These devices can also be embedded into furnishings or other types of objects. In addition, they can store and re-synthesize sounds by nuancing them using data transmitted by nearby sensors. The research project therefore targets the creation of a series of devices – some made from soft material – that will react in different ways to proximity and contact, movements, noise characteristics, and the progress of the game itself. The sonic behavior of the devices are designed in the spirit of games such as hide-and-seek and blind-man’s buff and also work well with a variable number of players in both ad hoc and rehearsed situations.

When the project was launched in November 2006, the WYSIWYG team experimented with a prototype ”blanket” able to sense how it is handled. During the presentation, eight people manipulated this photo-sensitive blanket to produce a spatial sonic landscape. In July 2007, dancers performed a semi-choreographed movement improvisation around a 20’ suspended “tapestry”and a 6’ “tablecloth” woven with conductive thread on a Jacquard loom by Joey Berzowska’s XS Labs.

With WYSIWYG, Sha and his team intend to develop other architectural-scale sensate cloths that function as agents co-performing with dancers and as image-bearing, kinetic surfaces in other performance contexts.

Dancer Marie Laurier with 20’ sounding cloth woven by Marguerite Bromley during Ouija workshop. © 2007 Topological Media Lab.

Custom electronics by Elliot Sinyor, McGill University. © 2007 Topological Media Lab.

David Gauthier with capacitive proximity sensor in the form of a bird woven from conductive fiber. © 2007 Topological Media Lab.

Critical studies of performance in responsive media spaces, based on installation-events designed as phenomenological experiments.

Livia Daza-Paris

Wet Petal	Visible Silence

Wolfgang Reitberger
Affective Dynamics in Responsive Media Spaces

Harry Smoak

Thick/N responsive space

Sha Xin Wei

A TGarden As a Phenomenological Experiment

Hubbub is an investigation of how accidental and non-accidental conversations can by catalyzed in urban spaces by means of speech projected onto public surfaces. Hubbub installations may be built into a bench, in a bus stop, a bar, a cafe, a school courtyard, a plaza, a park. As you walk by a Hubbub installation, some of the words you speak will dance in projection across the surfaces according to the energy and prosody of your voice. We'll capitalize on recognition errors to give a playful character to the space.

Hubbub is one application of TML research treating speech as a computational substance for architectural construction, complementary to its role as a medium of communication.

Success will be measured by the extent to which strangers who revisit a hubbub space begin to interact with one another socially in ways they otherwise would not. Hubbub is a part of a larger cycle called URBAN EARS, which explores how cities conduct conversations via the architecture of physical and computational matter.

TGarden is an investigation of how people make sense of and navigate in rich and dynamically evolving media spaces. Given the rise of ubiquitous computing and realtime media synthesis, we're anticipating the need for coherent yet supple ways for designers to create such complex interactive media spaces and for people to inhabit them.

In a TGarden space, visitors wearing instrumented clothing creates and modulate video and sound based on their gesture and movement. In effect, visitors write video and sound by their movement.

For 2001-2002, we concentrate on using wireless sensors on the body to track gesture. We have built a state evolution system that responds continuously to sensor statistics, synthesizes and marshalls media in realtime.

In TGarden spaces, we use a combination of costumes outfitted with sensors, video tracking, realtime sound and video processing, and gestural pattern tracking.

Research concerns include the design of continuously varying narrative spaces, how people improvise meaningful gesture, and factors of tangibility and coherence such as latency, temporal (musical) texture and rhythm. Our goal is to come up principles of design that should be useful for creating and inhabiting responsive media spaces. This research thread parallels a series of international productions in Europe and the United States.

Links:

SIGGRAPH2000 - New Orleans

http://sponge.org

http://www.f0.am/tgarden