>> You're not going to want to miss this special episode
of the AI Show coming to you with Microsoft Ignite.
Where we learn how the MET took it's images,
augmented it with cognitive services and
allowed all of us to find beauty,
even when it wasn't written down. Make sure you tune in.
[MUSIC].
>> Hello, and welcome to this special episode of the AI Show.
Coming to you from Microsoft Ignite, I've got a special guest.
Christina, how are you doing my friend?
>> Good. How are you?
>> Good. So tell us what you do.
>> So I'm a Program Manager on
the Cognitive Services platform Teams,
and I work on, among other things,
connectors within Spark and Cognitive Services,
and a bunch of other stuff.
>> Very interesting.
So we have a really interesting use case at the MET,
and how our Cognitive Services in conjunction with Spark,
with a bunch of other things, empower
them to do really amazing thing. Tell us about that.
>> Right. So the Metropolitan Museum of Arts,
decided to the start they're open access initiative.
What that meant, was they were going to release over
400,000 high-resolution images of art.
Like our pieces rarely see the light of the day,
and tell us a bunch more about us.
So we took all of that and then imbued it with
more intelligent annotations using
Cognitive Services to make it more searchable so that users,
visitors can explore the art collection in a more fun way.
>> That's interesting because most people don't know this,
but in the basement of all the greatest museums there's a lot of
amazing things that people would love to see but they can't.
But they had pictures of them and they use
these services in order to
get people to be able to look at them. Is that right?
>> Exactly.
>> So can you show us how this is done?
>> Yeah. So today's I'll go over
a little bit of how we imbued it with more intelligence,
and then we pushed all of this to Azure Cognitive Search so that
people could actually search and look at the art pieces.
>> All right.
>> So first we ingested the data from Metropolitan Museum,
and we see that there is a lot of these different metadata fields.
However, the problem was that a lot
of the metadata fields are pretty sparse, right?
So if we were to take
all the semantic information
solely based on these metadata fields,
then that would mean for some of our pieces,
we don't have that much information.
>> Right.
>> So that's where Cognitive Services came in.
So we ran Cognitive Services on Spark and we imbued
this little live intelligence that we can
glean from Computer Vision Service,
Text Analytics, and so on and so forth.
So we have all these high resolution images
of all these are pieces, right?
So we put it
through Computer Vision Service on Spark so that we could get,
are there people in it?
Are there faces in it?
>> I see.
>> Is it, like a lot of our pieces are pretty explicit.
Does it have explicit pieces so that we can
filter those out for younger audiences?
>> Sure.
>> If some of these pieces don't have a descriptive caption,
then we want to make this collection more
accessible for those who are visually impaired,
then we could do that with descriptive captioning.
>> So let me see if I understand.
Basically, tons of pictures,
you ingested them using Spark,
and as it was coming through,
you basically enriched those pictures with
additional information from Cognitive.
For example, there is a tree in this picture.
>> Yes. Exactly.
>> If anyone has ever used Cognitive Services,
you basically give because computer
vision the thing and it tells you what's in it.
>> Right.
>> There's also scores whether it's explicit or not, mature,
or whatever, and you're able to use
those to later enrich the data they were using?
>> Exactly.
>> Got it. Okay.
>> So we little that and then we found that there is
some interesting categories in these genres.
>> Were these categories generated by
the Computer Vision? Or were they-
>> Computer Vision, 100 percent.
>> So they generated these categories?
>> Right.
>> Okay.
>> So you could dial in to different categories from there, right?
So if I'm interested in images that are of people,
and then especially if I'm interested in
art pieces that are depicting
young people that I could go into that,
and then I can see art pieces that are depicting young people.
I can even see how their faces are,
and get some information on the faces.
Then this is just using Computer Vision,
but we could also invite
some more intelligence into the dataset using Text Analytics.
A lot of these art pieces come with curated description,
so so we could take that and then
name them to the recognition through their entities from them,
and then connect them to external references,
so that if you have
an art piece that's a picture of Ulysses S. Grant,
he's a former president, right?
So then who is Ulysses S. Grant?
Then you could go to Wikipedia to learn more about him,
and so on and so forth.
>> That's amazing because
you've gone through a bunch of images to some tiny description.
So it's something where you can
literally find exactly what you want.
I mean, usually when you're looking
at pictures, there's no way you'd be like,
"Give me all the young people from the 1800's."
>> Exactly.
>> But you could do this now with these kinds of services?
How long did it take them to put something together like this?
>> I believe this was a result of a three day hackathon.
>>Three days?
>> Yep.
>> Three days. Did they build
a website around this or an experience around this?
>> Yeah. So in order to do that,
we actually use Azure Cognitive Search.
>> I see.
>> So after we fit all these intelligent annotations,
we could actually use Spark connections to Azure Search
to take all this data
and then just put it to Azure Cognitive Search,
so that we can make a search engine out of it,
and then so people can go to a website called
Art Explorer to start searching for images here.
>> All right. This is the actual Art Explorer?
>> Yeah.
>> So you can just do cats,
and then it will give you images that are about cats.
Some of them really have cats in the titles,
but even for things that don't.
So if you search outdoor,
so obviously some of the images have the title, outdoor,
but then you get to these photographs of outdoor scenes,
but they don't have anything that indicates that their outdoor.
The actual metadata that we got from the Metropolitan Museum,
but Computer Vision could tell that these are outdoor scenes,
Pyramid of Giza, obviously an outdoor place.
>> Yes.
>> So you can get that information,
and then if you're interested in outdoor photography,
then you can just find it.
>> This is amazing.
Do we have this document anywhere people can see and use it?
>> So we actually did an AI Lab with this,
so AI Lab information is there.
Also, we made another civil [inaudible] called Jen Studio,
and all of that code is on GitHub.
>> That's amazing.
This is really cool because it's
hard enough that we have a ton of information,
but to make it accessible this way is really cool.
>> Yeah.
>> All right. Well, thanks so much for spending some time with us.
>> Thank you.
>> Thank you so much for watching.
We're learning how the MET,
took a bunch of images and made it so that everyone could
use it using Computer Vision,
Cognitive Services, and Apache Spark.
Thank you so much for watching.
We'll see you next time. Take care.
[MUSIC].