Hi everyone, my name is Ritam Sarmah
I’m a student working at UCLA HCI Research
and on behalf of my amazing collaborators from UCLA and Carnegie Mellon
I’m excited to present to you Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications
So, intelligent voice interfaces have exploded in popularity in the past few years
and products like Siri and Alexa have demonstrated that users appreciate certain advantages of the voice modality
namely, the convenience and efficiency of just telling a computer what to do
Now what happens if we consider voice input in the context of web apps?
Well, we get a lot of additional benefits, by enhancing accessibility for impaired users
and the expressivity of the pre-existing mouse modality
Despite all that, we currently aren’t seeing many voice-enabled web apps right now, and we asked… why?
Well, we think one of the major reasons is that implementing voice interfaces for web apps is challenging and time-consuming for developers
It requires writing new code and refactoring existing code
Many web developers don’t have prior experience with natural language processing, which is often a complex, specialized area
and there's a lack of consistent methods for creating multimodal interfaces
specifically for combining voice seamlessly with a prior mouse-based user experience
Which leads me to introduce Geno: a developer tool for adding voice input multimodally to existing web apps, without requiring significant NLP expertise
Let’s take a look at the end-user’s experience when using a Geno-enabled app like this calendar
In the bottom right we see a voice UI overlay, which Geno automatically adds for speaking voice commands
To change the calendar to a different view, we could click on it and say
“Change calendar to week view”
Now if we want to move an event to a new date, it’s as simple as hovering over the event and saying…
"Move this to next Tuesday"
And sure enough the event gets moved to the next week
We'll next look at the developer's perspective in using Geno to build the interactions we just saw
The high-level workflow follows three steps:
Specifying a target action, configuring the voice input, and optionally adding multimodal context
First, here’s a quick overview of the Geno IDE
From left to right, you can see a file explorer, code editor, and a browser app preview
Opening a project in Geno automatically generates all the necessary library files in the workspace to get the developer up and running
To create our first user interaction, or what we call an “intent"
we need to determine a target action to be triggered by the voice command
We can choose a pre-existing JavaScript function by just clicking on a circle next to it
or alternatively, we can choose a sequence of UI interactions by demonstrating them within the web preview
And when we're done, we see the same popover for configuring our voice input, which we’ll look at how to do next
To configure our voice input, we first add a unique intent name
and then add example utterances that the end-user can say to trigger the corresponding action
These examples are actually used as training data for a custom language model for the app
Now if we have function parameters, like in the case of moving an event, an event name and date
We can teach the model to extract parameter values from the utterances, by labeling them in each example
So clicking on an example opens up a secondary view where we can label entities with the corresponding parameter name
In this example, birthday party is the eventName, and April 15th is the newDate
Once we're done labeling all the examples, we just train the model
and now we have a fully functional voice interaction in our web app
We can now make this intent multimodal, which would allow a user to select one or more HTML elements via hovering or dragging as input
The selected element or its attribute values can be then passed directly to the target function as some parameter
For this example, we want to allow the user to hover over an event to provide the eventName
So, we first select a target parameter from the dropdown
click the crosshairs
and demonstrate the multimodal interaction in the preview
Geno then extracts the HTML attributes, which can be selected to be used as the parameter value
We evaluated Geno by using it to implement multimodal interactions for various scenarios, including:
searching for related news articles, manipulating complex 3D models, and adding songs to a playlist in a music player
We also asked 8 developers to implement four multimodal interactions across two web apps, the calendar and music player
Within an hour, 6 of the 8 completed all of the tasks, and we go into further detail on their experience in the paper
If you’re interested in more details and further discussion on our work
feel free to check out our paper as well as our GitHub code, which is accessible through this link
Thank you!