Computer-driven media are changing our environment, delivering images, sound and kinetic objects with ever greater density.   Given that increasingly complex information technology verges on the limits of intelligibility and manageability, we face the challenge of building and inhabiting our spaces in ways that can make sense to us individually and collectively.  How can we build rich responsive environments for shelter, sociality or play? How do people experience computer-mediated environments that now include not only virtual reality games and experimental theater, but also classrooms, airports and public spaces? In short, how can we build a world that is not complicated but rich? 

In order to answer these practical and conceptual challenges, the Topological Media Lab was established to study gesture and embodied use of hybrid computational-physical materials.   Dr. Sha and TML researchers investigate how we build, inhabit and use sensate or active matter. By this we mean combinations of computational and physical materials sensitive to environmental features or activities, responding by changing their form or appearance. We say material because the emphasis is not so much on objects or devices, but on continuous substrates. 

The experimental aspect of this work proceeds at two scales. The micro scale concerns topological responsive media, which includes time-based media and computationally-augmented fabrics.   The macro scale concerns the architecture of responsive media spaces, which includes augmented reality, sensor-based interactive environments, projected and ubiquitous media.

The experimental approach is based on a critical, theoretical project which treats the world as a continuous ontology.  Prof. Sha’s theoretical work explores the limits of discrete representation, finding alternatives to linguistic-semiotic analysis in the form of non-metric topological, dynamical, potential-theoretic and other material patterns.  This theoretical project is informed by a material and social phenomenology.  This investigation is substantively based in a fusion of computer science, science studies and critical studies of new media.

Intertwining scientific work with cultural practice gives meaning and context to guide the research. Prof. Sha and associates collaborate with fellow creators such as the sponge and FoAM art research networks to materialize these ideas as artifacts, installations and public conversations such as the TGarden responsive media spaces.

PROPOSED RESEARCH THREADS

We describe some of TML’s principal inter-related areas of research: gesture and performance, realtime media choreography, active garments, responsive media, hybrid architecture.


PERFORMANCE AND PHENOMENOLOGY OF GESTURE

We investigate what performance means at the fine scale of gestures inside a responsive media space. The approach is not typical of what is traditionally called performance theory, but synthesizes insights from activity theory, studies of material agency, user experience design, the study of writing, and other domains.

TGarden Projects.  We investigate what meaningful, compelling gestures are possible with sensor-augmented  experimental clothing modifying computer-generated fields of sound and video.

The series  of TGarden projects has already yielded a path-breaking hardware / software / garment / media system – TG2001 – that has been deployed under harsh performance conditions at Ars Electronica and V2 Rotterdam.   Since then, in 2001-2002, we have built a lab testbed for media choreography and have extended it to incorporate live synthesis of rich video transformations.   Its elements been extended in the txOom responsive spaces  (Italy, UK 2002) and a physical game system by other groups working in the public cultural sphere (FoAM, Belgium; Times Up, Austria).

(See videos of past TGarden installation experiments Siggraph New Orleans 7/2000, Medi@terra Athens Greece 11/2000, Ars Electronica Linz Austria 9/2001, V2 Rotterdam Belgium 11/2001, plus an extension, txOom, Future Physical 12/2002.)

Inspired by the on-going cycle of the TGarden responsive environments, we will focus on developing wireless and sensor technologies that work well embedded in clothing in socially marked ritual spaces.   We investigate how gestures can be mapped in compelling ways to  continuously varying fields of sound and video. We are particularly interested in how people collectively and individually habituate themselves to dynamically varying responsive media-fusion play environments without resorting to ordinary language. 

Some of the technical questions concern the mechatronics of body-borne sensors and sensate clothing, un-tethered gesture and body tracking, wireless systems. Some of the most interesting challenges lie in the realtime statistical analysis and interpretation of improvised gesture.

Based on the TGarden testbed, the Topological Media Lab is extending the continuous evolution state engine as an infrastructure that can marshall, hint, coordinate the synthesis of rich video and audio material in realtime response to people’s gestures.  At the same time, this authoring paradigm allows designers or  composers of responsive spaces to give a potential shape to the dynamics and the aesthetics without mapping them into a procedural programming language.  The key conceptual contribution here is to make the representations and transformations from topological dynamical systems usable in a fully controllable but not overly finely deterministic and metrized mechanism. The research is predicated on the conjecture that the “qualitative” but rigorous logic employed by designers of interactive media systems can be well-represented and even effectively computably represented by realtime systems using this topological approach. Preliminary results over the past year and a half encourage us to develop this approach into an authoring system for a greater range of designers and responsive media spaces. Of course interaction designers can control, or choreograph the media choreography via the authoring system, but an essential aspect of the TGarden is the distribution of agency to the inhabitant players and to autonomous computational or even material/physical processes for live, realtime expression. Usually this sort of relinquishing of control raises great chances of system failure due to unanticipated input, but our topological approach is also designed to produce qualitative effects robustly in the face of perturbations of input. Indeed, this sort of robustness plus richness of behavior is one of the strategic long-term research interests of the TML. In sum, our experimental approach to managing complex media logics and environments extends to the interaction designers as well as inhabitants a way to manipulate media using dynamical metaphors that are rich and simultaneously operational.

This work is substantively informed by deep expertise derived from experimental theater, performance and dance.
 

SOFTWEAR: RESPONSIVE GARMENTS AS EXPRESSIVE INSTRUMENTS
 
Another research project evolving from the TGarden project is the construction of sensate and active garments. The active garments research capitalizes on engineering advances in sensor technology, wireless technology and experimental textiles. However, beyond the wearable technology, this work depends critically on ideas from experimental performance, costume design and choreography, and most importantly, on live musical performance. 

In fact, this work capitalizes on the past century of expertise and  rich design intuition from the domain of live electronic music and electronic instrument design. We continue our collaboration with J. Ryan and M. Waisvisz and colleagues in STEIM and other institutions for electronic music and sound (e.g. CNMAT at University of California Berkeley, and Sony Computer Science Laboratory in Paris). This work also builds on collaboration between the Topological Media Lab and experts in the domain of active fabrics.  A recent result is the TCostume garment with sensate liner provided by Prof. S. Jayaraman that conducts signals from TGarden sensors along signal-conducting fiber embedded into the garment to the TGarden wireless computer.

In the domain of active textiles, we are collaborating with J. Berzowska at Concordia who is one of the pioneers in active material and aesthetic computation and media. With such techniques we propose to create a set of expressive, symbolically charged garments that can be tested singly or in conversational settings.  

This represents the first of several technical stages of the wearable instruments initiative:

(1) Embed current TGarden sensors and wearable gear using conductive fiber or cable on custom costumes.
(2) Experiment with alternate garment body affordances and constraints, designed with alternate sensors, gesture-tracking, maps to sound and video processes.
(3a) Adapt realtime visual/audio synthesis system to emissive, electronically addressable imaging fabric-based materials, and establish new designs for such materials.
(3b) Adapt realtime visual/audio synthesis system to kinetic, electronically deformable fabrics, and establish new designs for active materials.
(4) Stage integrated experiments with multiple people wearing active garments in responsive environments.

These garments are designed as instruments that can work alone, but come alive in most compelling patterns in the presence of TGarden-based environments. HYBRID ARCHITECTURE AND HABITATION OF URBAN SPACE

In this macro-scale research thread, we investigate how people build, destroy, modify and inhabit city environments using embedded computational systems. The first part of this study is social and historical, employing methods of field observations as well as insights from phenomenological and anthropological studies.  We intend to combine this work with insights of colleagues from the domains of urban design and architecture to design computer-mediated, responsive environmental systems for urban space. 

The HUBBUB research series presents an foray in this domain of urban responsive  architecture. As you walk through a Hubbub space, your speech is picked up by microphones, your speech is partially recognized and converted to text. Associate text is projected onto the walls, furniture and other surfaces around you as animated glyphs whose dancing motion reflects the energy and prosody of your speech. Hubbub is an investigation of how accidental and non-accidental conversations can take place in public spaces, by means of speech that appears as glyphs projected on  public surfaces. The installation takes its meaning from the social space in which it is  embedded, so its "function" depends on the site we select. Some of the technical issues concern realtime, speaker-independent, training-free, speech recognition; realtime extraction of features from speech data; multi-variate continuous deformation and animation of glyphs in open-air public display systems, such as in projection or in LED displays. We will investigate how embedding such responsive media as the Hubbub speech-painting technology as well as TGarden technologies into the urban environment can modestly support rich, playful forms of sociality. 

REFERENCES:

Hubbub, http://www.gvu.gatech.edu/people/sha.xinwei/topologicalmedia/hubbub Melting Pot or Fruit Salad: Experiment in Autonomously Producing Culture, in preparation for CODE volume, 2003.  (PDF) Pliant Research, http://www.pliant.org Resistance Is Fertile: Gesture and Agency in the Field of Responsive Media, in preparation for MakeOver volume, Configurations 2003 (PDF). Sauna, http://www.sponge.org/projects/m3_sauna_intro.html TGarden Media Choreography System, with Y. Visell and B. MacIntyre, 2003 (PDF). TGarden Research, http://www.gvu.gatech.edu/people/sha.xinwei/topologicalmedia/tgarden
TGarden Performance, http://sponge.org/projects/m3_tg_intro.html
TML, http://www.gvu.gatech.edu/people/sha.xinwei/topologicalmedia/
http://www.lcc.gatech.edu/~xinwei/