>> Search with pictures
on Bing and also learn how to build
skills based upon that search with my good friend
Robbie on this episode of the AI show.
Welcome to another episode of the AI show,
we're going to talk about visual search
with my friend Robbie. How are you doing my friend?
>> Good. Great to be here.
>> All right. Why don't you tell us what
it is that we're talking about today?
>> Sure. We'll be
working on in Bing is something we call visual search.
Basically, visual search is basically
uploading an image and searching on that image.
Basically, based on the content, what's in the image.
>> So, wait. Hold on. Because there's
something a little sci-fi to me.
You're saying, instead of typing in, like,
Gears of War because you want to play the game,
you upload a picture of it.
>> Exactly. Because there's a lot of things that
you want to know about the world that
you can't really express in words.
For example, there's this picture of this flower.
If I saw that on the street how do I
really try to find out what that flower is, right?
There's no way to really describe that in
words that'll try to get the right result.
So, what we're doing is being
able to just take a photo of it and
we'll find what that is by identifying the flower.
>> What kind of magic sauce are you
using for that? Can you tell us?
>> Yeah. We're doing a lot of stuff.
So, the first thing we're doing is we're doing
things like detection of
objects and entities in the image.
We're also using various DNN models to
recognize various entities in the image.
And they're also searching for
similar images across the Web to
find similar images that
provide us knowledge back about the original image.
>> That's amazing. Can you show us how it works?
>> So what we're actually releasing, sorry.
At Bing, we've released the visual search API which
allows developers to get access to all of
the content that we just kind of went through.
So, developers can go here and get
a various set of recognition capabilities.
For example, detecting animals,
detecting flowers, detecting landmarks.
And then the other cool thing is some
of the things that we detect
like fashion items like clothing and shoes,
we're actually able to find
similar products so you can buy online.
So, let's say I really liked your shirt,
I can just take a picture of it and find places to buy
that shirt or a shirt that looks just like it.
>> And this is an API that I
can use inside of my application.
>> Exactly.
>> That's cool.
>> It's part of the cognitive services API
which will allow
you to infuse intelligence and
AI into your products and services.
>> So, it looks like you have a lot of
awesome capabilities. What else can it do?
>> So, good question actually.
Well, what we actually announced is
the Bing Visual Search Developer platform.
So, what it allows is developers to
extend the capabilities of visual search,
to create their own new scenarios by
providing new knowledge or providing new actions.
So, for example, in this screen you
see you take a picture of a furniture table,
furniture outlet skill can provide you where
you can find similar products like that to buy.
>> I see. So is this something that, as a developer,
I would build then when someone searches for
an image then my skill would come up and say,
"Oh, you're looking at furniture. We sell furniture."
>> Exactly. Actually, the best way to do this is
actually kind of a demo and show you some skills.
>> Awesome.
>> So, here's a set of visual skills
that we kind of have here.
So, for example, in the first one,
it's an interesting fruit.
If you look at that, you probably
won't know what that is.
So, you just take a picture
and we have a skill, for example,
that provides you what
recipes you can make with that particular fruit.
Similarly, you take a picture of a landmark.
We can provide you various information
about it as a skill.
Another one is if you take a picture of a flower,
how do you take care of a particular flower, right?
And it provides that as a skill as well.
>> I see. I see. I see.
So how does a developer set this up?
>> Great question. So, I'll actually
go to building a skill for you.
So, you can go to basically
bingvisualsearch.io and start building a skill.
>> Cool.
>> So I'll create a skill that's very simple where,
given a picture of food, it'll
find where I can get that nearby.
>> Okay.
>> And I'll leverage Bing for this.
So, I'm just going to create a new skill.
So, I'm going to call it "Food
near me" and of course I'm making
it, "Finds food nearby".
>> So, as you're typing this in,
is this creating something that when
someone goes to Bing and does a search,
your skill will show up or
I'm trying to find where this fits.
>> Sure. So, where of
the Bing Visual Search API
is leveraged across Microsoft First Party Services,
it'll be exposed there.
Currently, we're exposing this in the Bing iOS app
and the future Bing and
other Microsoft First Party Services.
>> I see. So, if you build a skill like, let's just say,
you are a restaurant tour that has
many restaurants and you want to be able
to inject a skill based upon
what people look at or really want to,
they search for a picture of like a pie,
then it'll show and be, "Oh, you can go
see where we can buy pies."
>> Exactly. Exactly. So, I
just submitted all the info about my skill.
The next part is actually building your skill,
and we actually have three simple steps
and that's all that's required.
So, the first step is when should we trigger your skill?
So, we call this visual intent.
Visual intent is defined as what is
the most likely intent of
the image given the content of the image.
So, for example if the picture is of a glass,
it's probably about finding about
that drink or where you can find some eyeglasses,
or if it's a picture of a shirt, it's probably trying to
buy it or find similar products.
So, we have two options here.
One option is that you can always trigger your skill
or you can actually customize with visual intents.
So, we provide you a set of visual intents and
we're growing these constantly and improving upon them.
So, in this case, for the food one,
I'm going to actually look for food and drink.
So, in this case, when there's
a picture of a food item, it'll call my skill.
When there's no picture of food it won't call my skill.
So, it'll save my capacity and bandwidth for my skill.
Then the next part we ask is
that what data should we send to you.
So at Bing, as part of our API,
we recognize a lot of things.
So, we can send things like
annotations that are terms that
describe what's in the image.
We can send the raw image.
We can detect to various objects and send that
as part of the metadata we send to you or even OCR.
So you can kind of pick and choose what you want.
So, for in this case, I want Bing to
actually tell me what the food item is
because I don't know myself.
So, in this case I want Bing to
kind of tell me what the food item is.
So, I'm going to choose "Annotations".
The next and third thing we ask for is the service URL.
So, where are your code is actually hosted.
So, I already have my code and
a link but I'll just kind of brief through the code here.
So, the code here is basically very simple.
It looks pretty large just because there's
so many content, so many comments.
But if you look at it the
actual code to run a skill is actually very simple.
So, this is like the raw set
of information we need to run your skill.
And of course, no one
should code this way because you want
proper error handling and all those kinds of things.
So, in this skill what we're doing is
that HTTP request is
being sent to my code and I'm parsing
through the annotations metadata that's being sent to me.
>> Which is what you requested?
>> Exactly. And then based on the annotations,
I'm creating a response
back and then I'm
crafting what the preview tech should be,
the image URL, and the click-through experience be.
And then I'm just packaging that
back into a response in the body.
>> It's like a little card look and type thing.
>> Exactly. Exactly. So, I have a link for
this service and basically that's it.
You just built a skill.
And now we can actually start testing it.
So, I have an image of my dinner last night.
>> Nice. You're making me hungry.
>> So, I'm going to actually take
this picture and upload it.
And then the minute you upload it we start
running our API and
start getting all the information about this image.
So, this is the Jason response is actually being sent
to your service based on
the configurations you made earlier.
>> Cool.
>> So, you can use this to code against it,
check and debug and all sorts of things.
And then the next thing you can do is actually
click "Run" and this is where it actually
hits your code and
waits for the response and generates a little preview.
So, here you can see it created a little preview.
It recognizes that the food in the image is
fried rice and it said
search Bing for fried rice near me.
And then I can just click on this and it'll go to Bing.
>> That's nice and you can put
whatever you want in that card.
You might be able to deep link into, like,
a specific category of products in
your own website as part of the search results.
>> Exactly.
>> So, now I'm understanding.
This is enabling, like,
in a generic search engine to actually
provide specific customization based
upon what people are seeing,
not necessarily what they're typing.
Is that what you're doing?
>> Yes. Exactly.
>> That's amazing. Cool.
>> And I have one special skill just for you.
>> Oh, let's see it.
>> We call it the hotdog-or-not skill.
>> This is important. This is like we
could build an entire company on this.
>> We made sure this works
end-to-end before we shipped anything.
>> That's right.
>> So, here I could take
a picture of let's say, for example,
this pizza and I'm going to upload this and I'll run.
And then you can see it detected it's not a hot dog.
>> There you go. This is cool.
Now we got to put the hot dog in to make
sure it actually really works.
>> Exactly.
>> Nice.
Awesome.
So, where can people
go to get started on learning how to do this?
>> So, they just go to bingvisualsearch.com
and all the information,
documentation, everything is over there.
And then we also have one other feature
that we're releasing.
>> Let's see it.
>> Very soon, which is allowing you to
test your skill on the Bing iOS app.
So, it's very simple for you to just go around,
start taking pictures of things and see if your skill
triggers and get new ideas on how to build skills.
>> That's amazing. Like I had to build a skill like,
is that Seth or not?
So, people always are wondering around.
The world will know is that
really Seth. You know what I'm saying?
>> Exactly.
>> Would you use that skill?
>> Of course.
>> Yeah because of course.
Well, thanks so much for spending some time
with us, Robbie. This has been amazing.
We've been learning all about
visual search and how to build
skills on top of Bing Visual Search.
Thanks so much for watching.
We'll see you next time. Take care.