OK.
Hello everyone.
My name is Miro Mannino.
I'm working with professor Azza Abouzzied for New York
University of Abu Dhabi.
So again, like, today we're going
to present a paper called Is this Real?
Generating Synthetic Data That Looks Real.
And so let's start.
So data generation is an important tool
for a variety of users from teachers.
We want to teach statistical methods or data
science, software engineers, developers
who wants to demo their system.
And after this, user resort to synthetic data
in a variety of cases.
The first case is when access to real data sets is difficult.
For example, medical school records, income data,
et cetera, et cetera.
So the second case is when real data sets are not
in the right granularity.
For example, publicly available--
oh, sorry.
Publicly available data through initiatives
such as opendata.gov for example, are often aggregated.
And the third case is when real data sets are not
in the right scale.
Data owners often provide restricted access
to a small subset of a larger and richer first party data.
And the forth case is when a real data sets are incomplete.
A poor data sample might be missing
a key illustrative trends or a key field entirely.
It is important to note that users view realistic data
generation as a secondary, but essential to a primary task.
So for example, teachers primary goal
might be to expose students to data driven decision making.
So she decides for example, like an easy to learn
an effective tool for data generation
that can help her to focus on her primary goal, which
is building an overall lesson plan.
But the problem is however that realistic data generation
is a technically complex task with limited tools.
And user can manually code the task,
but that takes time and expertise of course.
And especially, if there are many statistical processes
and relationships that describe the data.
They can use tools like Mockaroo,
but they're very simplistic as well.
Or they can use the simulation tools,
but they need to understand how to describe data.
For example, using a Bayesian networks which is not trivial.
And finally, most of these tools do not
follow how users generate data, which
is through an iterative and interactive process
where they can incrementally specify, and instantly seal
their data being generated.
For this reason, we built Synner which
stands for a synthetic data generation.
And I will now introduce similar to a specific use case,
and with a live demo.
So imagining a social science professor
who wants to help students to study differences in retirement
age between US and Canada.
And she has this graph from public records
that describe the data.
But without samples for her students
to apply data driven method, so she quickly
decide to build samples that will lead
to these differences in income.
And here they can see the fields like country, age,
determine income, and country determined city.
She also asked name to her data to make
appear like it's coming from a real set of people.
So let's go quickly to the demo.
So you can see I'm going to--
OK.
You can see here that there is a top part where
you can describe all the fields you want to generate.
So here, we type--
for example, we wanted the first field to be name,
and we type name as a name of the column.
And here you can already see that Synner is already
suggesting the name as a possible way
to populate this field.
And if you click here, in fact is going to get to be field--
go to.
OK.
I'm going to generate as a name itself.
So here you see a preview of the data that is already generated,
and this is generated instantly while we click here.
And here in Instagram is showing the distribution of this data.
So we generate another other fields,
which is in this case city.
And here again, we see that is already suggesting.
You might want to generate this field as a city.
And here you see that depends on--
so it basically means that you can make name depends on
are the things in this case, there is nothing really.
It's like, not good to think like, name depends on city,
maybe.
But we can't-- we need to do country.
So we want to generate another field,
and make city to depends on this country.
But Synner provides here once we add
provides a way to create a new field, and in this case,
country.
And clicking here is creating automatically country,
and linking them together.
And the way Synner does those suggestions is basically
going through the database, and seeing all the relationships
between this database.
And this case was able to suggest
that you may want to create a country,
and link them together.
Of course, this is not hardcoded.
This depends on the current database.
Tomorrow if you haven't your own use case,
you can load more data and more relationships.
And this is going to work anyway.
So we want another field age.
And for age, we do our uniform distribution,
so the age and the slides was 1880.
And we generate basically age.
This is straightforward.
And then we want income.
So income, we-- if we were watching this lives again like,
it depends on both age and country.
So we want to highlight the difference between US
and Canada, and as well as the more is the age,
the more the income is going to grow.
So let's create this by using something
called a visual relationship.
So here, basically, you can define your age first
of all and income.
So in these slides, I'm not sure about the current values.
But it's not important to be strict right now.
OK.
So let's do those incomes from 10,000 to 80,000, age,
and we can throw the trend.
So we can do like the slides that we saw before.
And in this case, the age--
for example, for age 18, the value 10,000
is going to be generated all the time.
So we want to do some noises around it,
and the way it's in there is able to do
Gaussian noise around the visual relationships here.
And then we create a new case for a country Canada.
So this is basically like a switch in programming language.
And we have a different behavior for in case country is Canada.
And we have a different visual relationship
where again, the age and we do something different that
looks like the graph before.
And of course now you can see that countries are now
really only US and Canada, so we need to fix this.
And the way we can do this is by creating an expression,
like a filter.
And this is a Boolean expression.
We can say a country has to be equal to Canada or US.
But in this case, it's probably cleaner
just to use enumeration.
So we basically specified the two values we want to generate.
So Canada and United States.
We can even like create distribution,
so we may want to have more values coming
from US rather than Canada.
So basically meaning that when we have United States,
the income is going to have this behavior here.
Because this is the default case, it's not the Canada one.
So the data is going to be generated
by following this one.
If there is a Canada value, the other one is going to be used.
So this a very quick overview.
So let's go further, and--
OK.
So Synner employs four main design principles,
which I will go through in detail now.
And the first one is visual lifting.
This term means lifting of the mind specific language
into an isomorphic higher level visual language,
which is more natural for user.
So a good example of this in Synner
is the visual relationships, as you saw in the demo.
Where users can simply define how
two fields are related by sketching a trend,
and the data is generated by following that pattern.
And another one is supporting the clarity of specifications.
We specified that we want to have a certain distribution
between these two countries, but we did not say anything
about how to generate them.
Not the all green behind it.
Another principle is example driven interaction,
and Synner infers what the user wish
to generate from column labels or values manually inserted
into the data spreadsheet.
So from a few values inserted into the H-column for example,
Synner suggests a number of statistical distribution
you may want to use.
To avoid the illusion of precision,
and enhance the trust in the tool,
it's important to illustrate the user
that the generated data deviates from
their idealized specifications.
And the Synner visualize multiple possible outcomes
for a given sample size.
You can see here with a larger sample size
that the generated data is closer to the specification.
So what are the valuations we conducted?
We had two hypotheses in mind.
One is that the user are going to spend less time compared
to other methods.
And the other one is that the user
is going to be able to generate more realistic data easily
with Synner.
So we ask users to generate data,
three different data sets where users used Synner and Mockaroo.
And developers wrote data generation scripts.
And the full detail of the valuation of course
in the paper here, I'm just only showing you
one task, which is that children's growth record.
So for children growth record, we
asked user to generate age, sex, and height.
And height depends on age, and sex in the plot
as you can see in this slide.
And on the right, you can see that the graph
that each dots represent the time for task
competition for each user.
And the black bar is the main time to competition.
And you can notice how it was it's
generally faster to generate data using Synner,
and developers both higher than graduating student of course
took much longer.
But finishing the task by itself is not really enough.
It's also important to consider how
realistic data, like how realistic
is the generated data.
So but there is no really established measure
to measure this.
So what can we do?
We did something that reminds us of the Turing test.
So we ask people to observe random data,
and tell us some rules that anyone
can use to determine whether or not the data set is real.
For example, one rule requires that for each age group,
there is some height variation.
And the specification that generated a fixed set
height for each age failed this test for example.
So we find that data produced by the developer
satisfy many of the realism rules,
because developers were able to introduce
a lot of customization.
However, they were more prone to implementation errors,
as seen as the age distribution.
But in general, we saw that end user produced a more realistic
data with Synner.
We also conducted the qualitative evaluation
through a post study questionnaire.
User found Synner to be more expressive and more powerful
when compared to Mockaroo, and they
felt that they were able to generate more realistic data as
well.
So to conclude, we saw that the technical challenges
in creating tools to generate realistic data, we introduce
some the same principle that can help
creating powerful and easy to use data generation tools.
We introduced Synner.
And we show how with Synner not only user
were able to specify how to generate data more easily,
but also they were able to generate more realistic data
when compared to other tools.
And so also feel free this is a ready project
this is a live on our website.
And you can see here the link Synner.io the is also
open source.
You can load or expand for your own purposes.
Because this is like kind of a general purpose tool,
so you may free-- you may want to expand it as well.
OK.
Thank you very much.
Do you have any question?
Hi.
So very nice work.
So in your first example with the income distribution,
the teacher decided to make the data more realistic
by adding city.
But in fact, inadvertently created
data in which income does not depend on city, because there
was no dependents there.
There was nothing supportive there.
Now, you obviously can't prevent people from doing this.
But is there any way that you could
warn people of correlations or lack of correlations
that they are putting into their data set
that they may not be aware of?
Yes.
So this is for example, something
that we are doing right now.
So there is like warnings.
So if for example, if have name depends
on city, which is not possible.
Like, it doesn't make sense like, there
is going to be a warning saying, this relation doesn't
make sense.
And again, there are other warnings for relationships.
For example, cycles or like, maybe
like name that both depends on two different things
they're competing against each other.
So all of those you cases are in a borderline cases
where you need warnings for it.
And right now there are only a few of them.
So for cycles and stuff.
But there is going to be other warnings as well
for that of course.
I thought two questions or comments.
OK.
Sorry.
[INAUDIBLE] from Google.
Two questions really.
One is that, can you say more about the theoretical basis
of your model like other than saying
it's some sort of basic distribution model plus noise.
So I want to hear more about that for testifying the tool.
The second is that is Turing test applicable to problems
that are fundamentally kind of a natural human behavior.
Like, I can imagine judging speech or judging handwriting
would be a pretty good test, but for something like even
gesture typing.
We developed a model using minimal jerk,
and some statistical theories of human motor control.
But we ran into this problem that what Turing type of test
a good rule for judging the validity of the data.
Because fundamentally, humans don't do that in everyday life.
OK.
So to answer the first question of course
it depends on the generator.
So if I have some slides here as a backup.
So there are different generators in the backend side.
And of course if you want something more fancy or more
advanced, you can always expand and create your own generator.
So right now, there are only limited amount
of generators like domains and distributions,
but of course we are even working
on other generators that are having lots of data
is an input for example.
So this is up to you depending on your use cases.
This is more like a general user interface and framework for it.
The second question is like the Turing test.
So it's not exactly Turing test.
Of course, we just has Turing test as an inspiration.
Of course, it's hard even for human.
We found that it's hard to judge if data is real or not.
So our approach was more like-- instead of telling us
if it's real or not by looking at this telling us
if you-- what are the rules that you think they
could be useful for judging that.
So there's a different approach as well.
Well of course the Turing test is different of course.
And we can do one quick question.
OK.
[INAUDIBLE]
Go ahead.
OK.
Hi, Mary Beth CMU.
I'll make this quick.
More of a suggestion and a question.
There's a lot of implicit relationships in data,
like I noticed a guy named Humphrey Smith is living
in North Korea that seems surprising to me
that I saw on your demo.
There's these implicit relationships
that you can't really recognize computationally, but it really,
really important for realistic data.
And for me teaching my students.
In future tools, is there sort of a way
that you could have people encode
these sort of assumptions that are
embedded in sort of implicit signals in their data?
So those kind of relationships that depends on your database
at the end.
So the [INAUDIBLE] that the database, and the more
relationships like this you can do.
So right now, there are only few of them.
But of course in the future, like,
if you want to expand the database, you can always do.
You can always load more domain, and more relationships as well.
Let's thank our speaker.
I'm sorry.
We can discuss later.