Hi everyone. I’m Julia Cambre, and I’m excited to introduce our work on Firefox Voice, an open and
extensible voice assistant built upon the web. This work was done at Mozilla, in collaboration
with the Consumer Voice Products team. Mozilla’s mission is to “ensure the Internet is a global
public resource, open and accessible to all,” and when we think about the scope of that resource,
the amount of information that’s available through the web today is pretty staggering…
as of 2019, there were almost 1.7 billion websites estimated to be online globally.
And over the past several years, voice has emerged as an increasingly common way of interfacing with
the web: websites are beginning to offer the option to dictate search queries rather than
type them, and within the context of voice assistants, recent studies have found that
search and informational queries consistently rank among the most frequently used features.
At the same time, however, the number of voice apps and the information that's
accessible by voice assistants represents only a tiny fraction of what’s available on the internet…
and the number and types of voice experiences are also constrained by the fact that today’s popular
voice assistants are largely closed ecosystems.
So the guiding question that motivated our work was this: How might we help to make more web
content available by voice, and to foster an open voice ecosystem? As a first step towards
that goal, we built Firefox Voice, which is an open source, browser-based voice assistant.
To give you a sense of how Firefox Voice works,
I’ll demo a few of its features with some CHI-related browsing tasks.
Hey Firefox, what's the weather in Yokohama?
[Firefox Voice] "It's 53 degrees and clear."
Go to the CHI conference registration page
Play "Put That There" on YouTube
Hey Firefox, find my talk slides tab.
You can find full implementation details within the paper, but to briefly unpack some of what
you saw in the demo, users can initiate Firefox voice through a toolbar icon, keyboard shortcut,
or an optionally enabled “Hey Firefox” wakeword. The wakeword detection model runs locally in the
browser using Tensorflow.js, and was built by our collaborators at the University of Waterloo.
Once Firefox Voice is activated it listens for users’ speech,
and sends that speech to Google’s speech recognition service for transcription
via a proxy server. We then take the transcription and map it to a supported intent using a
simple slot-filling approach. Finally, we execute the intent’s corresponding action,
which in some cases, involves extracting and displaying an information card from Google,
and in others, performs actions on the user’s behalf through browser and web APIs.
All of the code for Firefox Voice is open source, and we were fortunate to have a strong amount of
engagement from the open source community, with 286 merged pull requests from 50 contributors…
and we’d love to have others fork and build upon Firefox Voice for their own voice projects.
Firefox Voice was the culmination of several years of research within Mozilla’s Consumer
Voice Products team, with dozens of different studies that informed the final version of the
system that you just saw. In the paper, we detail our iterative process of developing and deploying
Firefox Voice, focusing specifically on four main studies, including formative work through
a large-scale needfinding survey and a series of user studies with early working prototypes.
We also evaluated the Firefox Voice once it was deployed through system log data,
and by surveying users shortly after they installed and uninstalled Firefox Voice.
Between December 2019 and August 2020, Firefox Voice was installed over 30,000 times,
and had a total of 12,635 all-time active users. And to hopefully pique your interest
in reading the paper, I’ll highlight just one example from our findings.
This graph shows the number of daily active users from July 1st to August 31st 2020, where daily
usage initially was around 30 users a day, then increased dramatically to about 1000 a day, before
falling and leveling off about around 180. That spike corresponds to a week in which Firefox Voice
was featured within the new tab page on Firefox, which drove a large influx of new installs.
We see this as an illustration of some of the often unarticulated realities of
real-world voice assistant deployments, and in the paper, we discuss some of
the potential implications that it suggests in terms of improving discoverability and retention.
Please check out the paper and the code-base for more details, and thank you for listening!