The key to effective multi modal design is?

1. Antennae Sculpture 2. My shadow – a radio frequency 3. Studio environment housing equipment.
From “Frequency and Volume” an installation by Rafael Lozano Hemmer on display at SFMOMA, December 2012

My first project on joining [24]7 Inc was to design a multi mobile consumer application. What made it multi modal was the use of the speech and mobile web browser used together on a smartphone.

Creating this multi modal application provoked rich collaborations across parts of the company that had been recently acquired and were new to one another. The combination of late nights, adrenalin and frank conversations sparked an energy vortex that was hair raising, heart pumping and the very essence of Silicon Valley startup life at its finest.
As I tried to write up my notes about the affordances our design had created to share with our sales team, I was horrified to discover that the passion was not coming through on paper. Was it easier to design than to write about it? Was I too close to reflect on the enormity of what we had just accomplished? My writing fell flat where spoken word inspired confidence.

“The key to effective multi-modal design, I wrote, is to ensure each mode plays to its own strength. When combining modalities of speech and screen together, the screen is best used for displaying visually intense or data rich information, for example, choosing a date from a calendar. The voice channel, when employing continuous interactive speech is good for taking short cuts, for example, speaking a bank account number, than typing it using a mobile keypad… “

Unimpressed with my write up of our design, the team asked for a re-write, but nothing came to me. Seeking inspiration for my writers block, I went to the San Francisco Museum of Modern Art and found myself experiencing an alternative multi modal design experience.

Rafael Lozano Hemmer built a participatory installation that makes radio frequencies visible in the gallery. The installation is made up of a large antenna sculpture outside on the roof; a gallery, where visitors interact with a vast array of private and public radio signals and a studio environment housing technical equipment that works on its own.

In the gallery each persons shadow is detected by a computerized tracking system and their unique position and outline determines which frequency is scanned. Visitors use their bodies as antennae and tune into each radio station.  Moving your body left and right lets you tune into different radio frequencies. With people walking around the gallery together, each affecting a different signal, a cacophony of radios channels is heard. A voice in a sea of voices. Chaotic but not discordant.

Changing the volume involves walking back and forth making the silhouette of your body on the gallery wall change from big to small – loud to soft. The playful engagement, easy discovery and group dynamic put a huge smile on my face.

The installation, “Frequency and Volume” is one of several projects Rafael Lozano Hemmer has worked on since 1997. His participatory installations examine ways “the physical and public environment can be transformed into a data space in which intangible information is mapped onto a physical location.” This piece is a response to the Mexican governments shutting down of unregulated stations in indigenous communities in the states of Chiapas and Guerrero in 2003.

The key to effective multi-modal design, I can now rewrite, extends far beyond describing modalities and what they are good for. The key to effective multi modal design is to make the user smile. A smile born from the joy of playful discovery. A discovery that comes from participating with technology in community where one can be seen and heard and feel safe to do both.

Guidepad: design vision for a mobile travel application

Guidpad mobile application screen design

Mobile screens for travel application Guidepad

Founder Labs is a five week program focused on the first phase of launching start ups in the mobile space. In January 2011 I  participated in the program and collaborated with an engineer and product business lead in the design of a mobile application Guidepad.

Challenge: In only five weeks, the team met, determined a problem space and customer, prototyped a solution, sought customer insights and presented the findings and results to an audience of investors.

Response: Guidepad sought to reduce the stress of traveling for affluent travelers visiting destination cities in the USA. Available from concierges at high end hotels, Guidepad offered travelers a customized mobile phone with essential travel services, internet and phone. Above are screen shots from a use case scenario I designed for supporting a traveler safely back to their hotel in a city they may not be familiar with.

Response: To learn more about the my lean start up mobile work at Founders Lab, please read my blog posts that track the experience week by week.

Role: Designer

Year: 2011

ReQall: design vision for a mobile personal assistant

ReQall images from workshop

Scenes from the Reqall design vision workshops

Engaged by ReQall, a 4 year old mobile startup, to infuse the spirit of human centered design into defining the next generation of mobile personal assistant

Challenge: I set up and led a series of workshops with the entire international mobile team exploring the creation process for the next generation of service, informed by customer and team insights. The vision process included some of the following team activities which I led.

  • Sharing stories from the team and from customer insights, helped the team frame the context for design. The stories were about real people and their lives and served as inspiration opportunities, ideas and solutions.
  • Identifying patterns and themes arising from the stories, revealed overarching truths. The truths let us see the design challenge in a new light
  • Creating opportunity areas meant spending time on each pattern or theme and asking “How might we…” It was a stepping stone to idea generation
  • Brainstorming new solutions took the themes to the next level and were illustrated simply with words and pictures often on bits of card or the whiteboard. To keep the brainstorms real, we kept asking; How might we? Who will benefit from the idea? What is the value? How will payment be collected?


  • Generated opportunities, solutions and storyboards to inform prototypes for testing and feedback
  • United the team around a shared context for design of the next set of services
  • Fostered a generative mindset within the team
  • Led to solutions informed by genuine needs and connections to deep thoughts and feelings

Testimonial: “We brought in Sally to bring cohesion to our design process. In particular, Sally facilitated the team’s participation in the process by infusing human centered and emotional aspects into design – an area in which she excels. Her contributions are much appreciated by one and all on our team.”

Rao Machiraju, CEO, ReQall

Role: Workshop leader and facilitator

Year: 2010

Yahoo! Connected Life Design Vision

Yahoo! Connected Life Design Vision

Yahoo! Concept Diagram for the Connected Life Consumer Experience

Connected Life brought together three digital components of a typical consumers life – their phone, their television, and their computer. As creative director I led the user experience design teams to deliver the connected life customer experience extending Yahoo’s brand reach to become as ubiquitous as possible.

Challenge: Connected life consisted of Yahoo Go Mobile (aimed at mobile phones), Yahoo Go TV (for televisions), and Yahoo Go Desktop (for computers). With a Yahoo ID, users could connect the whole Internet to their mobile phone and Internet-connected TVs, as well as personal computers without using a browser.

Response: As head of user experience design for Connected Life, I built a world-class team of design experts to develop engaging personal and community-aware products in TV, desktop, broadband and mobile platforms.

The broadband and mobile teams had established products and revenue streams and could provide resources to develop the Yahoo! Go experience. The TV and desktop platforms were unproven businesses but needed a design vision which could demonstrate the value of the business.

Results: Using my experience in R&D and waterfall product development practice, I was able to build a multi disciplinary team of user experience designers capable of both pioneering new user experiences for TV and Desktop and evolving the existing businesses in broadband and mobile.

The investment in design paid off. In 2006 Yahoo! Go was unveiled to good reviews at the Consumer Electronics Show (CES)

Today, much of the Yahoo! Go vision has been released from service. However, the legacy of the early ground breaking work lives on through Yahoo Connected TV,  and Yahoo! Mobile.

Role: Senior Director, Product Design

Year: 2006

Yahoo! Go Mobile Design Vision

Yahoo! Go Mobile Design Vision

Yahoo! Go Mobile Design Vision

Yahoo has made its content available on mobile phones since the first wireless enable cellphones came to market in the late 90‘s. In 2005 Yahoo! developed Go Mobile, a Java-based client application, the crown jewel of mobile offerings.

Challenge: Under the design direction of Pamela Mead, several levels of mobile service were developed allowing users with different types of phone to have an experience of Yahoo! on their phone, whatever the make or carrier.

Response: The most distinctive service, Yahoo! Go Mobile featured a “game changing” interaction design with carousel navigation for browsing Yahoo! content.

Result: The service allowed users to get all their Yahoo! information and content services in one place. Yahoo! Go Mobile seen above, was deployed as beta software and performed well until the Go platform underpinning the service was closed in December 2009.

Role: Senior Director, Product Design

Year: 2006

Yahoo! Connected TV Design Vision

Yahoo! Go TV Design Vision

Yahoo! Connected TV Initial Design Concepts

Yahoo! Go TV, or Connected TV as it is called today, explored the opportunity to transport the Yahoo! user experience from the desktop to the living room. It meant building a design team who could develop a 10 foot user experience.

Challenge: The challenge of Yahoo! Go TV was to create a team of designers who could develop a go to market solution and a design vision that could attract new partners to collaborate and invest in an emerging business.

Response: Dr. Mike Mills led the design team who prototyped a series of short and long-range concepts. A short term go to market solution made entertainment services available on any PC-connected TV through a small downloadable application on the PC. The services included local and video search, movie trailers, movie times, TV shows, user ratings and weather, sports scores, stock data and news from My Yahoo.

Long term design visions were created as working prototypes to show how Yahoo! personal content could be integrated with live broadcast. Single point solutions called ‘widgets’ were developed for delivering the latest sport or news highlights. A flexible interface for navigating and filtering personal media content for photos and music with10 levels of zooming was also built.

Results: The go to market solution was demonstrated at the CES electronics show in 2006 to good reviews. The long range design prototypes were shown to business partners and succeeded in creating opportunities for new joint ventures.

Since August 2010, Yahoo! Connected TV has deployed widgets for TV’s providing customers with the latest updates on information from across the Yahoo! network along with Movies and TV shows on demand.

Role: Senior Director, Product Design

Year: 2006

Excite Mobile Application Design

Excite Mobile Application Design

Excite Mobile Application design

By 2000, mobile phones with Internet capabilities were coming to market rapidly. As part of its business strategy to extend customer reach and gain market share in the emerging mobile Internet market.

Challenge: As the mobile application designer, my task was to transform existing internet applications like email, news, entertainment and weather onto a limited-screen-space mobile phone experience.

Response: At the time, images on mobile internet phones were black and white and with screen sizes that could be as small as three lines of text. The work was highly creative despite the design constraints imposed by the hardware and software limitations.

Results: Of the many applications I designed, developing the first customizable internet enabled mobile phone home page in the U.S. allowed users to choose the mobile applications they wanted to use. The innovation gave Excite leverage to compete head-to-head with Yahoo! for placement of WAP services on the highly valued mobile home pages of carriers such as Sprint.

For Excite customers, the service provided increased freedom from their PC, now that applications were available anytime, anywhere and on any computer or mobile Internet device. The Excite, the service extended the brand to more customers, some of whom were accessing the Internet for the first time from their cell phones.

Role: Designer

Year: 2000

Kodak Picture CD Design

Kodak Photo CD Vision Work

Kodak Photo CD Design

Eastman Kodak was, at one time, the global market leader in camera film. During the 1990s, the rise of digital photography negatively impacted Kodak’s sales, and the company needed to find a way to recoup its losses.

Challenge: Before the Internet enabled the storage and processing of digital images, CD-ROMs bridged the digital gap by offering both traditional film processing and digital pictures. The challenge was to show Kodak how they could transform the marketplace for consumer photography by providing customers with a new way of working with their photos on a  CD-ROM.

Response: I developed the interaction and user experience specifications for the first working prototype of the Kodak Photo CD. Kodak provided the functionality, based on consumer insights, and Intel provided the technology. Product features supported storage, sharing, printing, and retrieval of photographs, as well as limited editing, slideshow display, wallpaper, and screensaver functionality.

Consumers would bring their CDs into national retail chains that offered Kodak Picture CD services and pay approximately $8.95 – $10.95 for two-day turnaround of their photographs.

Result: Designing the Photo CD helped Kodak to transform itself and the marketplace for consumer photography. The Kodak Photo CD proved to be an international retail success.

Role: Interaction designer with design agency Studio Archetype

Year: 1998

Apple India Mobile Healthcare

Apple Mobile Health Vision Work in India

Apple India Mobile Healthcare

The Indian government asked Apple to develop software to run on the Newton PDA that would permit rural healthcare workers to keep more accurate census and family planning records. I was part of the initial team who made many trips to rural Rajasthan to study the people, environment and healthcare system to inform the design of the product.

Challenge: Inaccurate capture of census and family planning data in rural India costs the government significant time and money. It makes planning for healthcare needs. Records are traditionally paper based and written in many different languages. Accuracy is lost when data is translated into English and transferred up the healthcare hierarchy.

Response: This type of research was ground breaking for Apple and the Government of India. There are no rules for developing products like this. My response was to assimilate what I saw and act as communicator to the team and beyond and explain the value of what we were doing and why it was important.

I set about collecting and analyzing all the official healthcare documents the nurses had to fill in, then had them translated into English. This activity revealed much about the wide spread number of Governmental programs in effect and would inform the information architecture of the software.

Writing and producing the video “Padma’s Story: A Day in the Life or a Rural Indian Healthcare Worker” helped our sponsors see the unmet needs of the 360.000 auxiliary nurse midwifes of rural India and our work to try and solve some of them.

Noticing the absence of electrical outlets in rural villages and field hospitals I designed a portable solar power panel to charge the Newton. The panel was sewn into a shoulder bag the nurse carried on her rounds.

Results: The project was ahead of its time and the problems we uncovered are multi layered, dynamic and just as relevant today as they were in 1994. The video and papers I published continue to be sited by students of mobile healthcare research to this day.

The software was completed and prototyped on the Newton 2000. Tests revealed it performed well in early field trials. The technology was transferred to India where its development continued, not in the healthcare market but for use by commodity dealers in the trading of tea.

Role: Design, research, information architect, documentary maker.

Year: 1994

Apple Big Table Vision Work

Apple Big Table Vision Work

Apple Big Table : Vision work for a collaborative work space

Big Table is large multi-user interactive table. It was an investigation into the implications of new technologies, based on real-world user activity, developments in display technology, and customer needs.

Challenge: Working with a small multi disciplinary team including interaction design, engineering and mechanical engineering, I lead a research investigation aimed at helping Apple’s publishing and multimedia markets by providing a larger working context for single and multiple users.

Technically, we looked at the possibilities and issues that arise when pixels become cheap and monitor geometries are no longer constrained by conventional CRT technology.

Commercially, we leveraged the Macintosh advantage inherent in the OS which permits multiple monitors to be linked easily.

Response: By building a working prototype (top image) for a large multi sided table, we were able to test ideas for how the table could support people in real-world activities. Some of the scenarios we prototyped and tested included:

  • Table as Public Work Space: People bring information to a meeting, often on paper, laptops, or mobile devices. A prototype for document sharing between devices and across the tables’ surface was developed. The center picture shows the team passing a document from a hand held device, onto the table surface across to another person at the table.
  • Table as Game Space: A table has a natural advantage because of its large display area as well as being an opportunity for an omni-directional display. To test this out, we devised a game that could be played from any side of the table. A four sided game of Pong was voted our most popular prototype with members of ATG.

Results: Through our explorations into hardware displays, multi user interactive work spaces, and integrated products, we touched the surface of opportunities for Apple’s markets in Multimedia and publishing. Apples’ Advanced Technology Lab was closed whilst the team were building the second generation of Big Table and the work was stopped.

Role: Design Manager

Year: 1997