The key to effective multi modal design is?

1. Antennae Sculpture 2. My shadow – a radio frequency 3. Studio environment housing equipment.
From “Frequency and Volume” an installation by Rafael Lozano Hemmer on display at SFMOMA, December 2012

My first project on joining [24]7 Inc was to design a multi mobile consumer application. What made it multi modal was the use of the speech and mobile web browser used together on a smartphone.

Creating this multi modal application provoked rich collaborations across parts of the company that had been recently acquired and were new to one another. The combination of late nights, adrenalin and frank conversations sparked an energy vortex that was hair raising, heart pumping and the very essence of Silicon Valley startup life at its finest.
As I tried to write up my notes about the affordances our design had created to share with our sales team, I was horrified to discover that the passion was not coming through on paper. Was it easier to design than to write about it? Was I too close to reflect on the enormity of what we had just accomplished? My writing fell flat where spoken word inspired confidence.

“The key to effective multi-modal design, I wrote, is to ensure each mode plays to its own strength. When combining modalities of speech and screen together, the screen is best used for displaying visually intense or data rich information, for example, choosing a date from a calendar. The voice channel, when employing continuous interactive speech is good for taking short cuts, for example, speaking a bank account number, than typing it using a mobile keypad… “

Unimpressed with my write up of our design, the team asked for a re-write, but nothing came to me. Seeking inspiration for my writers block, I went to the San Francisco Museum of Modern Art and found myself experiencing an alternative multi modal design experience.

Rafael Lozano Hemmer built a participatory installation that makes radio frequencies visible in the gallery. The installation is made up of a large antenna sculpture outside on the roof; a gallery, where visitors interact with a vast array of private and public radio signals and a studio environment housing technical equipment that works on its own.

In the gallery each persons shadow is detected by a computerized tracking system and their unique position and outline determines which frequency is scanned. Visitors use their bodies as antennae and tune into each radio station.  Moving your body left and right lets you tune into different radio frequencies. With people walking around the gallery together, each affecting a different signal, a cacophony of radios channels is heard. A voice in a sea of voices. Chaotic but not discordant.

Changing the volume involves walking back and forth making the silhouette of your body on the gallery wall change from big to small – loud to soft. The playful engagement, easy discovery and group dynamic put a huge smile on my face.

The installation, “Frequency and Volume” is one of several projects Rafael Lozano Hemmer has worked on since 1997. His participatory installations examine ways “the physical and public environment can be transformed into a data space in which intangible information is mapped onto a physical location.” This piece is a response to the Mexican governments shutting down of unregulated stations in indigenous communities in the states of Chiapas and Guerrero in 2003.

The key to effective multi-modal design, I can now rewrite, extends far beyond describing modalities and what they are good for. The key to effective multi modal design is to make the user smile. A smile born from the joy of playful discovery. A discovery that comes from participating with technology in community where one can be seen and heard and feel safe to do both.

The Future of Conversing with Technology

The Future of Conversing with Technology: Churchill Club Open Forum. Aug 2012. Palo Alto, California.

If you have ever wondered what computers might be like in the future and how we might interact with them, then read my notes from tonight’s Churchill Club open forum: The future of Conversing with Technology. A large crowd turned out to hear candid perspectives and insights from a diverse and lively group of visionary experts including Apple co founder Steve Wozniak and panel leader, Quentin Hardy, Deputy Technology Editor, The New York Times

Quentin posed several questions to the panel: What does it mean when we have a greater sense of intimacy with our machines? What shape will the industry look like as a result of voice – Nuance has taken the path of being the provider to many, Apple the provider of task fulfilling devices – Where will voice take us in the future?

Sheryl Connelly: Global Trends and Futuring, Ford Motor Company
Voice interfaces have the advantage of not having to adapt to the way we converse with natural dialog, this creates an opportunity for progress. Progress Ford call simplexity – it is how consumers will engage with information through voice technology at Ford.

“Using voice changes how you interact with your connected devices including the car.” From changing the temperature, to the radio station, voice becomes a means to a perpetual connection, through Bluetooth. It is a hands free and heavily amplified experience designed to overcome external noise.

“Young people don’t want a car as a status symbol, they want a productivity vehicle. Today there is market demand for the consumer to have connectivity in the car seamlessly. With the average Beijing commute time at 5 hours, having a device car that helps people be productive is key.”

Ron Kaplan: Senior Director and Distinguished Scientist, Nuance
Began his career as an engineer at Xerox Parc where he thought about how to create the illusion of simplicity for ordinary people from a really complex set of offerings. Parc went down two paths toward answering this question, the GUI and the CUI

The conversation user interface (CUI) aspired to use ordinary language to perform the actions people wanted to achieve. “This is using language like you would talk with a friend. The CUI understands what you want, when you say it, without saying everything. Perfection is not required, you just need to be comfortable getting the effect in a natural way, understanding 80% of what is said is fine. That’s about as much understanding as some conversations I have with my wife and the trash still gets put out on time.”

Steve Wozniak: Co-founder, Apple Computer
Today computers can engage the senses of touch, hearing and speech. Computers should save us from thinking, that is the model for voice interaction in the future. Speech alone has great possibilities because it is really robust, you can say things wrongly, say things that don’t make sense, but you can still understand one another. You can say things a lot of ways you can be understood and this is the way of the future. Engaging the senses, our devices must watch our face and know what we are thinking from our expression. In the long term, smell becomes important. We need to create our new best friend in our pocket.

Dan Miller: Senior Analyst and Founder, Opus Research
Security becomes interesting as you are carrying so much information on your device. Voice biometrics lets your unique voice become the thing that secures the device you are speaking to. A natural user interface way to wake up the machine using your voice allows only you to use the device. Voice is perfect for this. He would like a universal communicator to let him speak to his TV and thermostat. There are so many complicated home devices we don’t know how to use and the complexity undermines their utility.


So, wonder no more what computers might be like in the future and how we might interact with them. As Morgan Freeman and Martin Scorsese discovered in their commercials for Siri – it’s time to put down the mouse, pick up the microphone and learn to talk “proppa” to the “best friend in your pocket”. The question is, will it recognize your voice the next time you wake up with a cold? The Churchill Club open forum: the future of conversing with technology – nicely done.


Guidepad: design vision for a mobile travel application

Guidpad mobile application screen design

Mobile screens for travel application Guidepad

Founder Labs is a five week program focused on the first phase of launching start ups in the mobile space. In January 2011 I  participated in the program and collaborated with an engineer and product business lead in the design of a mobile application Guidepad.

Challenge: In only five weeks, the team met, determined a problem space and customer, prototyped a solution, sought customer insights and presented the findings and results to an audience of investors.

Response: Guidepad sought to reduce the stress of traveling for affluent travelers visiting destination cities in the USA. Available from concierges at high end hotels, Guidepad offered travelers a customized mobile phone with essential travel services, internet and phone. Above are screen shots from a use case scenario I designed for supporting a traveler safely back to their hotel in a city they may not be familiar with.

Response: To learn more about the my lean start up mobile work at Founders Lab, please read my blog posts that track the experience week by week.

Role: Designer

Year: 2011

ReQall: design vision for a mobile personal assistant

ReQall images from workshop

Scenes from the Reqall design vision workshops

Engaged by ReQall, a 4 year old mobile startup, to infuse the spirit of human centered design into defining the next generation of mobile personal assistant

Challenge: I set up and led a series of workshops with the entire international mobile team exploring the creation process for the next generation of service, informed by customer and team insights. The vision process included some of the following team activities which I led.

  • Sharing stories from the team and from customer insights, helped the team frame the context for design. The stories were about real people and their lives and served as inspiration opportunities, ideas and solutions.
  • Identifying patterns and themes arising from the stories, revealed overarching truths. The truths let us see the design challenge in a new light
  • Creating opportunity areas meant spending time on each pattern or theme and asking “How might we…” It was a stepping stone to idea generation
  • Brainstorming new solutions took the themes to the next level and were illustrated simply with words and pictures often on bits of card or the whiteboard. To keep the brainstorms real, we kept asking; How might we? Who will benefit from the idea? What is the value? How will payment be collected?


  • Generated opportunities, solutions and storyboards to inform prototypes for testing and feedback
  • United the team around a shared context for design of the next set of services
  • Fostered a generative mindset within the team
  • Led to solutions informed by genuine needs and connections to deep thoughts and feelings

Testimonial: “We brought in Sally to bring cohesion to our design process. In particular, Sally facilitated the team’s participation in the process by infusing human centered and emotional aspects into design – an area in which she excels. Her contributions are much appreciated by one and all on our team.”

Rao Machiraju, CEO, ReQall

Role: Workshop leader and facilitator

Year: 2010