>> You're not going want to miss this episode of the AI Show.
We look at what's new in image captioning, human parody goodness.
We're going to see how it's done,
we're going to see how you can do it. Make sure you tune in.
[MUSIC]
>> Hello and welcome to this episode of the AI Show.
We're going to talk about what's new with image captioning.
I've got a special guest.
Varsha, why don't you introduce yourself,
tell us who you are and what you do.
>> Yes. I'm Varsha,
Program Manager on Azure Cognitive Services team.
I work on pre-built computer vision offerings
like object detection,
image tagging, and image captioning.
>> Fantastic. Let's start first with people if they don't
know about what Microsoft has to offer with computer vision.
Let's start there. What does Microsoft
have to offer with computer vision?
>> Sure. Let's start with computer vision service.
Cognitive services vision offerings gives you access to
advanced algorithms that process
images and returns information based on visual features,
whether it's tagging content images
or detecting objects, identifying entities,
or even actions between these entities,
essentially helping our customers extract insights from images.
>> Sorry, you were saying, and what else?
>> Image captioning is part of the computer vision service,
which is something we'll dive into today.
It's designed to let you automatically extract
content and generate human readable image captions.
>> It's like we're the same person,
I was literally going to dive in.
We're talking about image captioning specifically.
You mentioned a little bit about what it was,
but what is image captioning?
What is it that we offer? Let's start there.
>> Yeah. I'm excited to share
the recent milestone we established with image captioning.
Our research team developed a new approach to image captioning,
and this breakthrough technology has helped us achieve
human parody on nocaps benchmark dataset.
Nocaps is novel object captioning at scale.
This is roughly used as an indicator of
advancement in capturing algorithms.
The images that you're seeing on the right has been
pulled out of the nocaps test dataset,
and the caption generated is through our AI model.
Seth, to be completely honest,
it took me a while to realize that
there's a table behind the waffle cone there.
It's pretty impressive.
>> That is amazing.
Look, I'm a computer vision person.
The level of classifying images
and telling if it's a cow or a horse,
I'm pretty good at that thing.
But image captioning that's this good,
how do we do that?
Can you give us a look inside at how we're getting
this human level captioning?
>> Yeah. Let's start with the process of
image captioning and why it's such a complex challenge.
The illustration that I'm showing here shows
the translating images into
words and the complexity associated with it.
Consider for a moment what it takes to visually
identify and describe something to another person,
and what if this person does not have
access or can see the object that you're trying to describe.
Every detail that you capture through your human eyes matters.
How does the machine decide
what information is important and what's not?
You need to know exactly what everything is,
where everything is, and what it's
doing in relation to other objects.
Well, also understanding what is the foreground
and what information is important and what isn't.
You really need to understand what's going on,
and a machine needs to be able to summarize this and
also describe it in natural language description.
>> That's really cool. I mean,
can you go back one slide
because you need to know every little thing in there,
and then you didn't know how to arrange
the words so that it makes sense.
Like if you said dog, cat, table,
that's not really a good sentence,
but knowing where everything is is actually really cool as well.
Sorry, I interrupted you. I just wanted to point that out,
that it's actually a hard problem.
>> For sure. Even with the fine tuning
we do generate multiple instances of
what we think as a human natural language sentence
and with the confidence score.
Customers are able to pick it up in terms
of what localization and what tags are more important to them.
>> Awesome.
>> Moving onto how these image captioning systems are designed.
They're typically trained with datasets that contain
images paired with sentences that describe these images.
It's a set of captioned images.
In this training set,
there's a combination of COCO dataset which is
about 80 object classes with caption images,
and there's also Open Image dataset which is
about 600 classes with just object tags,
there are no captions associated with it.
What makes this nocaps challenge even more challenging is,
it tests how the model is able to
describe so that the novel unique objects,
they are present in
the testing image but not in the training dataset.
In this example here,
the COCO object classes, for example,
like dogs, bench, or umbrella,
they're images in the train dataset
with captions and descriptions.
Whereas, for Open Images that you're seeing,
whether it's goat or dolphin,
it's just object tags,
there are no captions present.
>> Sorry, keep going.
>> But in the right, you see how well the model performs
on even out of domain object classes like dolphin.
>> Every class of thing that it can look at, obviously,
it needs to know about that and it's been trained on
these Open Image dataset as well as the COCO dataset.
Is that how it gets knowledge
about the actual things that are there?
>> Yeah. With the COCO dataset there are descriptions associated.
You're understanding the semantic coupling
between the different tags;
between the dog and the bench,
the dog is sitting on the bench,
or the child is sitting on the couch.
Whereas, with Open Images where it just tags the present,
there's very little information that
the model gains from just the tags.
There's no semantic coupling,
there is no dolphin in the water.
But as you can see,
we validly test this model
that out of domain knowledge is equally good.
A dolphin is swimming close to the ocean,
so the water coupling is happening with
the novel unique object that the model is seeing.
>> That's cool. I feel like I want to see some examples
of things that it's captured. Do you have anything like that?
>> Yeah. For sure, you're reading my mind.
>> Yeah, I see. It's almost as if we looked at
this and talked about it together before. Almost.
>> Almost. Let's take a look at a few examples
that we tested both with
our previous model that was released in 2015,
and with the latest human parody model.
As you can see the older model
gave us a caption of close up of a plant,
and now it's a close up of a wheat
in a field which is much more accurate.
>> Yeah. I mean, that's pretty amazing.
>> Yeah. Even here,
the older model gave us an update
of a blue shirt or a man wearing a blue shirt,
and now we can see its surgical mask.
It's a lot more contextual and in specific domain.
>> This is really cool because, I mean,
back when I was a programmer I would get
data that's in a database and with numbers,
with even strings, you could do stuff with text.
But with images it was hard to actually
figure out what's actually going on an image.
Now with this, you can totally just do it, which is pretty amazing.
Do you have any you can show us?
How do people take advantage of this kind of functionality?
>> I've few more examples.
>> Let's bring it on, let's do it.
>> We can jump to this one actually.
As you can see, the model is extracting
more of these unique objects present in the image,
and generating in many cases
more accurate description of the images.
>> This is cool.
>> This is one of my favorite ones where
AI model is not just detecting
objects present in the image but
also conveying the intent behind it.
The new caption generated is a person making bread,
which clearly the intent
and the insight that you want to decipher from the images.
>> Yeah. If I squint,
it does look like a person cooking hotdogs on a cutting board.
>> Yeah.
>> But now when you look at it, it's excellent,
you can see that there is intent behind a person.
It's pretty cool that it's capturing that as well.
You show me so many good pictures,
now can we take a look at how can I use this today?
>> Sure. I'm sorry. Let's start here then.
Here I'm showing the quick demo of how
the API can be integrated into your application.
To access Azure Cognitive Services API,
you need the Azure subscription.
You can create one for free using a free pricing tier.
Review the subscription, and create
the resource in specific deployment region.
Once you have Azure subscription,
create a computer vision resource in the Azure portal,
and get access to your key and endpoint.
You will need the key and endpoint
for the resource you create to connect your application,
to the Computer Vision service.
You'll be placing your key and endpoint in the application code.
Let's copy them over.
Now, let's pull up my Jupyter Notebook to see this in action.
Here I'm going to offer public services and system libraries.
Let's run that through.
Once that's imported, we'll add the subscription key and
Azure endpoint. Running that through.
Here's the interesting part.
The API can take in a URL and also a local image pack.
Seth, I'm going to put you on the spot here.
>> Let's do it.
>> Bring up a couple of Bing images.
What do you think of this one,
should we go ahead with this?
>> Sure.
>> We're copying the URL of the image and pasting it here.
Now, let's run this through.
Let's check what our image captioning generated.
Briana Banks holding the chart. Pretty accurate.
>> I don't even know who Briana Banks is.
>> Now, you know.
>> Now, I know.
>> Sorry, go ahead.
>> Fantastic. I just think this is great.
Do we have any other pictures?
>> Yeah. Let's try out a few more actually.
Do you want to give this a go?
>> Yeah, let's do it.
>> Copying the URL of this picture now.
Let's change the URL part.
Perfect.
Let's run that through and pull off.
Group of people and fireworks.
>> Let's try one more. Let's search for
Seth Juarez. Here, I'll spell my name.
Let's see if it knows who I am. Computers are funny.
Just, S-E-T-H and J-U-A-R-E.
There we go. Let's see if it finds a picture of me.
If I'm doing something crazy.
>> We do have a picture of you exactly.
>> Yeah, it do. Old school picture. Let's see what we got.
It's obviously we going to break the computer.
I'm sorry, I broke cognitive services. Let's try it.
>> Let's try this one more time.
It did actually, that's interesting.
Let me open up.
>> How about the one with me talking to some dude?
Go back one. Back one.
The one to the right of that.
Because that's the one to the right.
There you go. Try that one. Let's try that picture.
Let's see if it knows me or this guy.
Because maybe I was talking to a really famous guy
and I just had no idea.
>> Let's do it. I see what's happening. Sorry, give me a minute.
>> Oh, I didn't get the whole thing.
>> Yeah, I was just adding too big URL.
Let's try this.
>> Perfect. JPEG.
>> Not true.
I wish you don't know what's happening.
>> Yeah, I'll tell you. Go at the end,
and then there's a single quote you missed.
At the end of the string. Hit there, "Single".
There you go. Now, let's try it.
>> Perfect.
>> Here we go.
>> Men sitting at the table,
looking at a laptop.
>> Yes.
It knows that I'm sitting at a table, looking at a laptop.
Look at that. How cool is that?
This is cool. It seems pretty easy.
You set up the service,
you put in the key, and you just
start sending pictures to the thing.
It can be a URL or it could be
something you push up local. Is that right?
>> Yeah, exactly.
>> That's awesome. What else is new with computer vision?
Do you have anything new that we should be thinking about as well?
>> Yes. We've had a ton of updates this fall.
We've 3.1 version of computer vision APIs generally available.
We've updated SDKs for Python,
Java, C#, and JavaScript.
These leaders AI capabilities are
integrated with Intel Microsoft products,
and available to our customers using Office products.
>> I didn't think about that. Office is
using these things internally right now?
>> Yes. The next slide
I'll show the integrations that are
currently in our Office products.
>> Let's take a look.
>> The enhanced image captioning model
is already in our production service,
and is available in latest version of our Computer Vision API.
The service provides a flexible deployment option
enabling developers to use
this capability to both improve
accessibility in their own applications, and inter-phases.
The AI capability is already being used in Seeing AI,
Microsoft's Assistant mobile app for visually impaired.
A user can take a snapshot of their surroundings,
and Computer Vision Describe API will return
the description to help the users understand the scene,
and participate in the conversations.
Image captioning is also being used for
tasks like creating all text for images,
and this is available through
our Microsoft Office products like Office,
Outlook, Word, and PowerPoint.
>> This is really cool.
Where can people go to find out
more to get started on using this good stuff?
>> To get started, we have a bunch of resources available online.
You can visit Azure Cognitive Services portal or
directly go to Developer Documentation on Microsoft docs.com.
Look through API references,
concepts, quick-start guides, and tutorials.
If you have questions or feedback,
you can also contact us through
Azure Cognitive Services customer support.
>> Well, this is absolutely amazing.
I've enjoyed looking at this.
Like I said, I love computer vision,
is one of my favorite things.
It's cool to see it actually generating captions,
and doing all this good stuff.
Thank you so much for spending some time with us, my friend.
>> Thank you.
>> Also, thank you so much for watching.
We're learning all about the new image
captioning inside of computer vision.
Thank you so much for watching, and
hopefully we'll see you next time. Take care.
[MUSIC]