[MUSIC PLAYING]
HANNAH HIGHFILL: Hello and welcome to Google I/O 2022.
Thank you for joining our developer session,
"Building for Everyone, The Importance of Representative
Data."
In this session, we'll cover a few things--
how data representation carries an elevated significance
in AI and digital products.
We'll discuss a few helpful practices
and resources developers can use to address
data quality and representation bias in their projects.
And we'll introduce Task Mate Beta,
Google's experimental crowdsourcing tool
as a concept to further improve data quality
and representation.
My name is Hannah, and I'm a user experience researcher
working on Google's Next Billion User initiative.
As part of Google's Next Billion User initiative,
our team has a focus on making technology helpful, relevant,
and inclusive for people new to the internet.
To learn more about Google's NBU initiative, please
visit nextbillionusers.google.
To that end, one of the lessons that we learn over and over
again is that data quality and representation largely
determine the success of our digital products and AI
systems.
This is true as measured by metrics as diverse
as performance, fairness, robustness, safety,
and even scalability.
While these shortcomings happen early in the development cycle,
the impacts of poor data and representation
on model performance are often opaque and delayed.
And these effects compound into major negative impacts
downstream.
Paradoxically, for AI researchers and developers,
data is often the least incentivized aspect of the work
and is viewed as operational relative to the lionized work
of building novel models and algorithms.
This is true even though, intuitively, AI developers
understand that data quality matters, often
spending inordinate amounts of time on data tasks.
In a recent study, "Everyone wants to do the model work,
not the data work; Data cascades in high stakes AI,"
we found most organizations fail to create or meet any data
quality standards, undervaluing data work vis-a-vis model
development, and failing to define or meet
any standards of representativeness.
This is especially important in markets
where users are coming online for the first time, where
the model of one user, one phone, one account
is not standard, and the assumption
of continuous connectivity doesn't hold.
This research has also documented
how inattention to data needs causes
negative downstream effects, resulting
in technical debt over time.
Negative performance resulting from inattention to data
is also widely prevalent, affecting
92% of AI practitioners that we spoke with.
So what can be done to proactively address
these downstream impacts?
We're going to hear now from Ben Hutchinson, who researches
data ethics inside Google, for some best practices around data
collection.
BEN HUTCHINSON: Thank you, Hannah.
My name is Ben and I'm a software engineer on Google's
Responsible AI team.
To help avoid unexpected consequences later on,
it's critical to prioritize high-quality, representative
data collection from the start.
The first step to building representative data sets
is knowing your users.
Since the world is incredibly diverse,
data practitioners need to be clear from the onset
of the specific geographic and demographic populations they
intend to serve and ensure they have a plan to source data
which reflects their intended audience.
You may start by asking yourself,
is my data sampled in a way that represents my users?
For example, will the end product be used for all ages,
but I only have training data from senior citizens?
Is my data sampled in the real-world setting?
For example, will it be used year-round,
but I only have training data from the summer?
After having a solid understanding of the user base,
developers should consider the following leading practices
to source data that is representative
of the real world.
Set explicit goals around which region of the world
you want your data to represent.
Share these goals with your project team.
Source data directly from the respective region or audience
that you intend to serve so that your model and product better
understand and reflect local nuances.
Once you've collected your data, use
exploration and visualization tools,
such as Google's Know Your Data, in order to check for biases.
And, if you do find biases, make sure to document them
so that they don't cause nasty surprises for your end users.
Yes, data collection can be a tedious, time-consuming
process.
And it is just one step in a product's development.
However, as Hannah has pointed out, it's a critical step.
And it can have significant implications for the success
or failure of your end product.
It's important that developers give the data collection
step the respect it deserves and ensure
it reflects the make-up of the user base
you intend to serve to mitigate any negative downstream
impacts.
Another tool developers may want to consider
when sourcing representative data are general crowdsourcing
tools, like Task Mate.
Crowdsourcing tools are particularly helpful when
developers need to quickly create high-quality custom data
sets.
However, in some cases, to get the right expertise,
you might need to seek out more specialized tools.
If a crowdsourcing platform is used,
it's important for data practitioners
to be upfront in identifying themselves
and their organizations, fully disclosing
the purpose of the task and how the data will be used.
This is especially critical when data might contain sensitive
or identifying information.
Finally, paying crowd workers fairly for their work
is not only equitable but enables
sustainable relationships between taskers and task
providers.
This way, data contributors have all the information
they need to meaningfully decide whether a specific data
task is right for them to participate in.
And here to speak more about crowdsourcing as a solution
for creating high-quality, representative data sets
is Mike Knapp, co-founder of Task Mate,
Google's experimental crowdsourcing platform.
MIKE KNAPP: Thanks for the introduction, Ben.
Hi there.
My name is Mike Knapp, and I'm a product manager on Google's
Next Billion User initiative.
Because we know from our research
that it's really hard to create diverse and inclusive data
sets, we've been experimenting with a new crowdsourcing
platform called Task Mate.
The goal of this experimental product
is to enable developers to reach previously
hard-to-access, diverse crowds with a focus
on unrepresented regions and languages.
Businesses can customize their task
to suit a large range of use cases.
Some examples include data collection, labeling,
and field work to improve representation and ML
fairness in products, conduct surveys, and more.
Meanwhile, taskers can use the app
to complete tasks and earn money easily.
Let's look at a practical example
of where Task Mate has been used by the Google Research team
to expand the language capability of Google products,
such as Maps.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
- [NON-ENGLISH SPEECH]
[END PLAYBACK]
EMILA YANG: Hi.
My name is Emila, and I'm an interaction designer
on Google's Next Billion User initiative
and cofounder of Task Mate.
Task Mate is currently in an experimental beta testing
phase in three countries--
India, Mexico, and Kenya.
Let's now look at a selection of features on Task Mate.
It's simple to create a data collection
project on Task Mate.
You don't need any code to get started.
You can use a UI-based visual editor
to design a screen-by-screen workflow for your taskers.
There is a set of premade task steps, which
range from showing a map to direct taskers
to a location, recording audio, taking photos,
to viewing multimedia and answering questions.
You can set up these steps in any order you like
and customize them with your instructions.
These steps are designed to simplify and streamline
the experience for taskers and make the task easier
to get right.
Furthermore, Task Mate is integrated with Google Sheets,
allowing you to quickly enter or export your data
and enable quick pilots to test your task designs.
Once you've created and piloted your task,
Task Mate provides you tools to help scale up your project.
You can utilize our API to create
a more seamless integration and automate creation
and management of tasks.
Task Mate also enables you to set criteria
for the taskers you want to reach,
including regions, language proficiencies,
and demographics.
If you already have your own taskers,
you can add them straight to the project also.
We know quality can be a big challenge when creating a data
set, so Task Mate has created a variety of quality control
features.
There is peer validation, where you can set up
consensus on questions, or you can send the results
from one task to another person to validate.
We also have automated quality checks
for things like audio recordings that have too much background
noise, photos that are too dark or blurry,
and GPS checks to ensure field tasks
are done at the right location.
There are also automated privacy features,
like face and license plate blurring for photos
taken inside the Task Mate app.
For taskers, we've tried to make the experience
simple and enjoyable.
Our mobile-first approach creates
accessible opportunities for people
to do tasks and earn anywhere.
It enables a large range of tasks, including field tasks,
as well as tasks you can do from home.
To see all the current features on the platform,
you can check out taskmate.google.com.
We've also got a big roadmap ahead
and we're constantly working on delivering
additional improvements.
Next, we're going to hear about a field data collection
project run on Task Mate to develop an agricultural data
set in Kenya.
[MUSIC PLAYING]
ANNALYSE KEHS: The mission of PlantVillage
is to work with smallholder farmers
to help them adapt to climate change
and mitigate its effects.
JAMES MUGO: We have an application
we put into farmers' farms.
And they are able to go into the field
and they're able to diagnose their crops themselves.
ANNALYSE KEHS: PlantVillage combines
machine learning models and technologies
with human expertise.
One of the main challenges we face
is validating what's happening on the ground in comparison
to what we simulate is happening with our modeling team.
We use Task Mate to set tasks for users from here in the US
to specific regions in Kenya.
Users go to a predetermined location,
take a picture of the field, submit it, and then
earn income from that.
Today, we have collected over 600 images with Task Mate.
JAMES MUGO: Being paid for the tasks is a great benefit.
DONNA SARINGI: When you have Task Mate,
you can have some money that can be used during tough times.
ANNALYSE KEHS: Task Mate is directly
helping our farmers by, short term, providing them
additional income.
And then, long term, we're able to validate our AI model
results and then reach more people
with this highly specific advice.
JAMES MUGO: With Task Mate here, we
are talking about technology coming in to help the farmers.
We are going to look at the patterns and give solutions.
MIKE KNAPP: Today, we've talked about how data representation
carries an elevated significance in AI
and digital products, particularly
in unrepresented geographies and communities.
Building inclusive products requires a foundation
of representative data.
Google recognizes this important issue
and is experimenting with a crowdsourcing platform called
Task Mate to create high quality, relevant data sets.
Task Mate aims to tap into a diverse tasker network,
provide flexible tools and intelligent quality
controls to help you build more inclusive products.
Now, while we're still in beta, we
have a lot of really exciting things planned on our roadmap
and we'd love to understand your unique requirements.
So if you feel passionately about crowdsourcing
and want to participate in our beta program,
you can request access to Task Mate beta
by registering your interest at taskmate.google.com,
which is also linked in the session resources.
When you submit your inquiry, note
that you attended this session today
in the Additional Comments field.
If you're interested in learning more
about the importance of representative data
or accessing resources to help mitigate data
bias in your own projects, please download our session
developer guide, linked in the session resources
to the right of your screen.
Google's Next Billion User initiative
aims to create a more inclusive internet for everyone
everywhere.
We invite you to watch some of the other amazing NBU sessions,
"What can we learn from the internet's newest users"
and "Africa's booming developer ecosystem," on-demand
at the I/O website under the NBU tag.
Learn more about Google's Next Billion Users initiative also
by visiting nextbillionusers.google.
It's been so great to speak with you
today about these really important topics.
Thanks for watching.
[MUSIC PLAYING]