The key to effective multi modal design is?

1. Antennae Sculpture 2. My shadow – a radio frequency 3. Studio environment housing equipment.
From “Frequency and Volume” an installation by Rafael Lozano Hemmer on display at SFMOMA, December 2012

My first project on joining [24]7 Inc was to design a multi mobile consumer application. What made it multi modal was the use of the speech and mobile web browser used together on a smartphone.

Creating this multi modal application provoked rich collaborations across parts of the company that had been recently acquired and were new to one another. The combination of late nights, adrenalin and frank conversations sparked an energy vortex that was hair raising, heart pumping and the very essence of Silicon Valley startup life at its finest.
As I tried to write up my notes about the affordances our design had created to share with our sales team, I was horrified to discover that the passion was not coming through on paper. Was it easier to design than to write about it? Was I too close to reflect on the enormity of what we had just accomplished? My writing fell flat where spoken word inspired confidence.

“The key to effective multi-modal design, I wrote, is to ensure each mode plays to its own strength. When combining modalities of speech and screen together, the screen is best used for displaying visually intense or data rich information, for example, choosing a date from a calendar. The voice channel, when employing continuous interactive speech is good for taking short cuts, for example, speaking a bank account number, than typing it using a mobile keypad… “

Unimpressed with my write up of our design, the team asked for a re-write, but nothing came to me. Seeking inspiration for my writers block, I went to the San Francisco Museum of Modern Art and found myself experiencing an alternative multi modal design experience.

Rafael Lozano Hemmer built a participatory installation that makes radio frequencies visible in the gallery. The installation is made up of a large antenna sculpture outside on the roof; a gallery, where visitors interact with a vast array of private and public radio signals and a studio environment housing technical equipment that works on its own.

In the gallery each persons shadow is detected by a computerized tracking system and their unique position and outline determines which frequency is scanned. Visitors use their bodies as antennae and tune into each radio station.  Moving your body left and right lets you tune into different radio frequencies. With people walking around the gallery together, each affecting a different signal, a cacophony of radios channels is heard. A voice in a sea of voices. Chaotic but not discordant.

Changing the volume involves walking back and forth making the silhouette of your body on the gallery wall change from big to small – loud to soft. The playful engagement, easy discovery and group dynamic put a huge smile on my face.

The installation, “Frequency and Volume” is one of several projects Rafael Lozano Hemmer has worked on since 1997. His participatory installations examine ways “the physical and public environment can be transformed into a data space in which intangible information is mapped onto a physical location.” This piece is a response to the Mexican governments shutting down of unregulated stations in indigenous communities in the states of Chiapas and Guerrero in 2003.

The key to effective multi-modal design, I can now rewrite, extends far beyond describing modalities and what they are good for. The key to effective multi modal design is to make the user smile. A smile born from the joy of playful discovery. A discovery that comes from participating with technology in community where one can be seen and heard and feel safe to do both.