>> You're not going to want to miss
this episode of The AI Show where we
look at the version 2 of Form Recognizer.
There's some amazing things
in there, you're not going to want to miss it.
[MUSIC]
>> Hello and welcome to this episode of The AI Show,
where I am happy to introduce again,
Anna Roth is here. How you doing my friend.
>> Great. How about you?
>> Fantastic. For those that don't
remember what you do, why don't you remind us?
>> Yes, I'm a Program Manager on the Cognitive Services Team.
I work on Form Recognizer and ACR.
>> Fantastic. For those that don't know
anything about Form Recognizer,
why don't you get us up to speed?
>> Absolutely. Form Recognizer
is a cognitive service and it's designed
to let you extract structured content from all kinds of forms.
What you get a Form Recognizer are key value pairs
from an image of a form or a PDF.
Here for example, I want to extract things like the name,
the social security number,
first his name, social security number, address, etc.
You can see that I can pull that out.
It's a key-value pairs, so I can,
put it into a database or some RPA system or something like that.
>> I've heard RPA, what's an RPA system here?
>> RPA stands for robotic process automation.
You can think of it as a way to automate
coming clerical tasks that you might otherwise have to do by hand.
For example, rather than manually doing all the data entry,
you could scan a form,
do an extraction and check back and say,
okay, was that extraction correct?
>> Next question. Again, for those that haven't seen this before,
is there a restriction on the kinds of forms?
What kind of key-value pairs does is it pull out?
Just general knowledge here.
>> Absolutely. With Form Recognizer we have some pre-built forums,
but also what we call custom forums,
and so that's the ability to actually train
your own custom model
to recognize the content that you care about.
In that case, there's all kinds of different forms,
different businesses, group business processes might have.
With Form Recognizer, you train a model to recognize
your specific forms into key-value pairs that you want to extract.
>> In essence, it's really any form that your business uses,
you can pull or use this software form?
>> Exactly. I mean, that's the goal.
>> The next question is, so that's
a really good introduction to Form Recognizer.
The next question is, what's
new for those that are already familiar?
>> Absolutely. We've had a ton updates this summer.
First of all, we have v2.0 is now generally available,
so its the first time we've taken Form Recognizer
to GA. Two, we have new SDKs.
We've updated SDKs for.NET, Python, Java,
JavaScript, big enhancements to table extraction,
table detection, support for global currencies,
support for Azure Gov,
as well as enhanced security features
like customer-managed keys and private endpoints.
>> There's one more thing you are going to tell me about, right?
>> That's right. Yeah.
>> We also have the Form Recognizer, v2.1,
Public Preview now available,
which has a bunch of new features.
The first is language support,
we've gone from just supporting English to supporting Dutch,
English, French, German, Italian, Portuguese, and Spanish.
Then layout, which is the part of Form Recognizer,
it's a pre-built model for text and
tables extraction, will support Chinese.
>> These are all cool things.
I mean, for those who don't know,
they send me something and they're like we
want to talk about this new cool things.
I knew a Preview was coming up.
For each of those languages,
you mean that it will recognize
the key and the value for those languages?
>> Yes. I knew languages were really adding new OCR support.
Now, you can train a custom model for any of those languages,
and you have to build a train to recognize
key-value pairs across those languages.
>> All right. Sorry, I interrupted you,
but I love languages and so this is interesting to me.
Tell me about all the other things.
>> Absolutely. We support now also checkboxes and selection marks,
like little checkbox in your form,
both in layout and train with labels.
>> Sorry. I keep interrupting.
The other question I have, does this work
with actual handwriting as well?
Because when I see a checkbox,
I literally think of me and my pen just like either putting
an x or an actual checkmark and that works.
>> I'm glad you asked to us. No worries about the interruption.
Because that is a key feature actually, Form Recognizer,
is we have a very high-quality handwriting OCR models,
so we're able to recognize both printed and handwritten text.
>> The x or the checkbox or the checkmark?
>> That's right. If someone's handwritten a little x in the form,
that will work just as well as if you have a printed checkmark.
>> Awesome. That was the last interruption,
before I do it again, please go ahead.
>> For sure. We added Model Compose.
That lets you compose multiple different models that you've
trained so that you can call them with a single ID.
What that means is we'll then do a classification step and try and
figure out which of your models it is at prediction time.
Support for friendly model names,
new pre-built model for business cards,
new locales for pre-built receipts,
and then big quality improvements across the different operations.
>> Cool. So this is the next time.
Model composes is interesting to me
because I'm not sure I quite understand,
because you said there's a way to actually train
for your own kinds of forms.
If I have five different kinds of forms,
I have to call each one and I
have to know in advance what form it is.
Does this make that requirement go away? If so, how?
>> That's exactly what
Model Compose is designed to help solve for.
With Form Recognizer, you train a custom model,
basically for each template or for
each layout that you are trying to recognize.
Let's say you do have five different forms and you want to
recognize your five major variants of your form.
You would train five different models.
But maybe you don't know because your customers are
submitting this via an online process or you have multiple,
you might be given different forms at once,
you might not know which are the five forms
you've actually received when you have
a form you want to predict on.
What this lets you do is,
you can compose all five together,
send us a form and say,
"Hey, this is a form,
please tell me what the content is,"
and we'll actually do a classification.
Figure out which of the five forms it is,
and then give you a key-value pairs for that exact form.
>> Okay. If I'm understanding right,
and I'm not saying this is what you're doing,
but basically you have some logistic regression or something or
some other classification algorithm that looks at
the form and then tells it which actual model to use.
It's like a machine learning model
before it does the machine learning model perform recognition?
>> Yes, something like that.
>> That sounds awesome. I'd love to see it in action.
>> Absolutely. Let's take a look.
I'm going to show,
I think actually most of the things that are
new in the 2.1 Preview.
What we're seeing here is our FOTT tool.
This is an open source tool for labeling data,
which is usually the first step when people
are creating a custom forms model,
although it's optionally, you can do
everything using the APIs if you prefer.
But for the demo, it's a little easier to show here.
I'm trying to train
a new custom forms model to recognize this form,
which is the IRS form 8821.
What I'm going to do, I previously created
a couple of tags that I want to recognize.
Now, I'm going to go and label this form with these tags,
so let's take a look.
>> As you're labeling,
because I know this is very compelling television,
what is this program you're
labeling things with? I've never seen it before.
>> This is FOTT.
It's like an open source tool,
companion sample tool to Form Recognizer,
so that you can have a labeling interface if you need one.
>> I see. This is something that I can take and put into
my own business as part of my online software for,
we get a new form, I can mark it up and it
goes into the system kind of thing?
>> Exactly. It's on GitHub, you can rescan it.
We've had customers do that
where they'll just take it and rescan it,
or put it into an existing UX experience.
What I've done while we've been chatting,
I've gone through and actually labeled
all of these key-value pairs.
You said I would select one of these and I would say, okay,
that's the taxpayer, selected this text,
and said that's the appointee.
I'm giving examples and then going to train
on so that we can have our custom models.
So these are the key-value pairs that I want to go and
extract from this form
and that includes in this case the checkmarks.
So it says here this example is unselected,
selected, selected, unselected.
Now, I went ahead earlier just because of how
compelling the television is and labeled the rest of these forms,
so you don't have to watch me label every single one.
Now that we have five labeled examples,
we're ready to train our model.
>> You only need five examples to train the thing, right?
>> That's exactly right.
>> By the thing, I mean Form Recognizer.
>> Five is good to start.
You may want to increase depending on the complexity of the form,
but with five you can start training your model.
I'm going to give us a model friendly name.
I'm just going to call this the AI Show,
actually we'll call it form 8821 so
I know what this is and I'm going to hit "Train".
Now, I'm training a new custom model that is specific to
the key-value pairs that I just labeled
I get an estimate of the accuracy of my model here.
Now, let's see Model Compose in action.
Previously, in addition to form 8821 I also
went and labeled and trained
a different template or credit card authorization template.
Now, what I'm going to do is I'm going to select,
you can see here's the model ID that I just trained for form
8821 and here's the credit card authorization
and I'm going to hit "Compose. "
We'll call this new composed model,
we'll call this the AI Show Model.
Now, I'm composing these two together so I can submit
either the credit card authorization form or form
8821 which is a form that actually authorizes
viewing tax forms to one model ID.
This AI Show model so you can see that means it's
a composed model and it should be able to
extract the content from either form now.
>> Basically, let me make sure I understood the steps,
the first step you did was you train the form
8821 by labeling five, well, you labeled one,
but labeling five create the model called it form
8821 with the new feature with the nice model name instead of the,
I think that's a GUID .
The second thing you did is you said,
Okay there's another form that we had let's just put those
together and then it did its magic; is that right?
>>That's exactly right.
>> First question. How many models can be composed together?
>> As of today is up to 100.
>> Holy cow, that's pretty cool.
>> The second question is how long
does this inference take to run this thing?
>> Sure. It's a couple of seconds,
maybe on the order of 8-10 in
most cases. Should we see it in action?
>> Let's try it.
>> First, I'm going to grab one of our form 8821s.
>> This is a new form or is it one of the older ones?
>> It's a new form and let's go and click "Run Analysis."
Once again, I'm doing this here in this tool,
but I could also have downloaded this Python script so
that I could do this programmatically.
>> Got it.
>> We've got our results let's take a look here.
We went ahead and extracted the content from this form.
So we have our taxpayer here, our appointee,
our CAF number, etc., and all of these key-value pairs that we
previously trained have been extracted.
You can also download this we can see
the actual JSON was outputted.
So like Martha Stewart and
I actually already did this I can see this here.
[inaudible] we'll give you
the structured output so we go and extract our key-value pairs.
>> Can you pause right there so I want to make sure people
see this because when I figured this out,
because I did something with, not the
version two but the version one,
the bounding box basically tells you exactly
where on the file where it is,
the text or the value that it found, right?
>> That's exactly right.
>> The other question I have. Oh, sorry, go ahead.
>> You getting the key-value pairs, you getting the location.
We didn't show it earlier but you're also actually
getting the underlying table structure.
So there's a bunch of tables on
that form and that's being extracted for you too.
>> Then the other question that I had is when I
think of bounding boxes I'm usually thinking
of image files you were showing PDFs.
What other files can people use with this?
>> We support PDFs as well as a number of
common image formats, jpegs, PNGs, etc.
>> Basically if you take a picture of a piece of
paper that maps to a form it will still work?
>> That's right. Yeah.
>> That's pretty cool.
The other question I have is when you
submitted this particular form to
that model was that the composed model that you submitted it to.
>> That's right. Let's grab an example of our other form.
So you can see here we had our 8821, now, I'm actually
going to grab and we're going to do analysis on a different form.
So this was our credit card authorization form,
so here, we have our credit card authorization form.
Now, I'm going to actually run analysis here.
Once again, we're using our AI Show model
going to run this analysis.
What I expect now is that the key value pairs that are specific
to this form are going to be extracted.
You can see here the first thing we get is
this doc type so we've classified,
that we think that this is
the AI Show model credit card authorization form
as opposed to the 8821 which is correct.
We also get a confidence score here,
pretty high confidence this is not our 8821 which is right because
this isn't the same form this is our authorization form.
Now, you can see all of
these key-value pairs have
been extracted that are specific to this form here.
>> Including this signature which is what
goes back to actual handwriting?
>> That's right. Yeah.
>> That' really cool.
People can download all of this stuff and try it out?
Where can people go to find out
more get the source code for this stuff?
>> That's a great question.
We have a bunch of resources available online.
So you can go to this page to
get top-level information about Form Recognizer.
We have our documentation page which includes links to
the Open Source tool I showed earlier
as well as quick starts, ACK's, etc.
Some documentation on our OCR/Read API
if you just want to extract text.
As well as contact us either through
Azure support or we have this e-mail address.
>> That is fantastic. Well, this has been amazing.
Thank you so much for spending some time with
us and it's always a pleasure to have you on.
Again, thank you for watching and learning all about
Form Recognizer V2 and 2.1 preview
with something called Model Compose which seems pretty awesome.
Thanks for watching and we'll see you next time. Take care.
>> Thanks so much.
Bye bye.
[MUSIC]