>> In this episode of the AI show,
join us to hear from Neta Haiby,
Principal Program Manager on
the Microsoft Azure Cognitive Services Team.
Neta will show us how to extract
data from document and forms using
OCR and form recognizer. Make sure you take a look.
[MUSIC].
>> Welcome to the forum recognizers session.
In this session, we will extract data from
documents and forms with OCR and form recognizer.
Let's start with OCR.
The read API is
an off-the-shelf model train with millions of documents.
It's available as a managed service and as a container.
It extract text from documents printed in handwriting,
and also extract texts in the wild from images.
Our new Computer Vision API v3.0 is now generally available.
We also added support for the following languages, Spanish,
German, French, Italian, Portuguese, and Dutch.
Forms are everywhere.
Invoice application forms, tax forms,
bank forms, medical forms,
construction forms, every company has forms.
Even in these challenging times,
various scenarios require data extraction
from forms, unemployment forms,
employees sickly forms, loan application,
mortgage applications, COVID-19 relief applications,
clinical forms, and more.
Extracting data from these form today is mostly manual,
which requires long processing cycles.
Form recognizer enables you to extract
text key-value pairs and tables from your forms.
The input to form recognizer is document,
PDF or images, and the output is structured texts,
key-value pairs and tables.
It enables you to automate the process of
extracting data from your forms.
Form recognizer is comprised of three parts.
Layout, extract text and tables,
custom tailored to your form and extract text,
tables, and key-value pairs,
and pre-built, which are pre-built model pre-trained by us.
Let's start with layout.
Layout, extract text and tables from your documents.
It's an off-the-shelf model,
available as a managed service.
To extract text and tables,
all you need to do is send your document to the layout API,
and it will extract the text and tables,
and output is structured JSON output.
As an example, let's try and extract data from
the unemployment insurance weekly claims news report.
This is a weekly news report released
every week by the Department of Labor.
It's a multi-page PDF with lots of
data that is released as a PDF document.
To extract this data,
text and tables and enable analyzing the data,
we will use form recognizer layout API.
Let's see a demo on how to extract this data.
In this demo, we'll see how layout
extracts all the data from this document.
This document is the news release of
the unemployment insurance weekly claims,
which has 10 pages.
Some of them have text,
some of them have tables,
and what we want to extract is
the big table with
all this information so we can analyze it later.
All we need to do is go to our layout.
I created here a notebook that caused the layout API.
You enter your Form Recognizer Endpoint.
This endpoint is from the resource you
created in the Azure portal, and it's key.
We call here the layout API.
So you can see we're calling layout,
analyze. Let's run this.
Our new analyze Layout API is an Async API,
so you will also need to get the get results.
So let's scroll the get results.
This caused the gets results with the operation location
ID returned from the analyze layout.
This will run for a second on these 10 pages,
and when it's finished running,
let's print the output.
So we can see that it extracted,
and let's put them side-by-side.
We can see that it extracted all the text in the document.
So the connect, it starts from
the right top corner connect with the URL,
transmission of materials, unemployment insurance weekly claim.
So all the text is extracted,
but let's look for the visualization of our tables.
So we'll scroll down over all the text.
Now, we can see that we've also found several tables.
So we can see a table in page 4 was extracted.
Let's go to page 4, a table
for page 4 weekend, April 28th, was extracted.
Several pages before and the table we were interested in,
is the table that start on page 5,
and we can see that it was also
extracted with the first header as none,
the next header is initial claims during weekend, April 18.
Then again, none.
Then insurance unemployment, none and none,
and then the first row which has all the details state,
advanced prior week change,
advanced prior week change.
We can also see that,
and all the corresponding data in the relevant rows and columns.
So form recognizer will extract the data with row 0, column 0,
row 1, column 1,
and it will also tell you if it spans
multiple rows or multiple columns.
The output we'll get from form recognizer is a JSON output.
Let's view this as a JSON file, and let's scroll up.
So we have several parts to the results.
One is the page result which includes all the tables.
Select only the tables we wanted.
That's the table on page 5.
So this is page 4, page 5.
So we can see column index 1,
this spans on multiple rows.
So this one, the first column here spans on three columns,
the first row here spans on three columns and then we can
see that it was returned as column index 1 span 3.
It's bounding box so the text within it,
and the row is zero.
The next column is column index 4,
it also spans three columns.
It's row index 0 and it's the next title,
unemployment for week enabled.
Its bounding box, and also it
refers to the elements in the read results
where all the text was extracted if you want to get
the confidence of the words that were extracted.
So you have all the data column index 1,
row index 1 bounding box, the text.
So we have all the data to push it into a SQL database or an
Excel or any other place you want to do
analyze and do a calculation on this data.
So in a few seconds we were able to extract all the data
and all these tables from this 10-page document.
We saw how we can extract text and
tables with a layout form recognizer functionality.
Let's now look how we can train a model
to your forms with the custom functionality.
Form recognizer includes a custom option,
where you can train models specific to your forms into your data.
There are two available options.
One is train without labels,
which includes two steps,
the train step and then the analyzed step.
When training without labels,
no human labeling is required.
It's especially useful when you have a variety of foreign types,
and you want to quickly train a model for all those form types.
Only a handful of documents is needed to train without labels,
and to get started with form recognizer.
You do the train,
and then you can analyze documents.
But today we'll focus on train with labels.
In train with labels,
you first provide human inputs
to extract the values you care about,
and you label the forms,
then you train, and then you analyze.
Again, only a handful of
sample documents is required and needed to train a model,
and you can get started with just five forms.
You have the ability to create a continuous feedback loop by
adding additional forms to improve
the accuracy of the model over times.
To process mortgage application and loans,
lenders need to extract data
from lots of different types of forms.
With Form Recognizer custom service companies can train
models to automatically extract the data they need from forms.
Let's see how to train and analyze a document with
Form Recognizer custom service by
training a model and then analyzing a document.
To demo this, we will use a request
for transcript tax return form,
which is usually required for loan and mortgage applications.
To label the forms, we're going to use
the Form Recognizer sampling labeling tool.
The sample labeling tool is available as an open source.
It's also available as a tryout site
in our fott.azurewebsites.net location
or is a Docker container that you can pull and
deploy locally one Azure Cloud instance.
I already created a project here,
started and created a project
here for the form that we were looking at before.
I already labeled the name, the SSN,
the signature of the spouse,
another SSN, and the address.
Let's also label the form type and signature here.
So we're going to create a new tag and call
it form type and we're going to create
another tag and call it spouse signature.
We're going to label. So we select
the form type and we click on the "Form
type" to label it and we select
the spouse signature and we click on it.
Notice that the forms are already OCR,
so everything that's highlighted are things that you can label,
and we're going to label five forms to train a model.
So bear with me a few seconds,
we're going to label the five forms, three more.
Now that we can label printed, handwritten,
forms can be tilted,
rotated, and this is our last one.
Now, that we labeled five forms,
let's train a model.
So we go to the train and we click on the train.
This train button calls the Form Recognizer API with
your endpoint and the blob storage location
where the training files are,
and the label and OCR file.
You can see that with the five forms that we just trained,
we already got a model accuracy of 95 percent and we can
also see the estimated accuracy
for each one of the labels that we trained.
Let's analyze a file we didn't train on.
Let's select the file to analyze and let's run prediction on it.
What prediction does is it sends
this file to our Former Recognizer,
analyze form API with a model ID and the file and it
identifies what are the values and what
are the label that we want to extract.
So we can see that it extract John Doe has the name, SSN1,
the signature that we labeled,
the SSN2 that we labeled, the address,
the form type and the spouse signature.
We can see that it gave confidence to each one of
the labels and the values that were extracted.
The output from Form Recognizer is a JSON output file.
The read result, which is
all the text that was extracted from the document,
its bounding box and confidence, the paid results,
which are the tables that was extracted from the documents
if they were tables and the document results,
which are all the key value pairs that we just labeled.
We can see that the doctype is
custom form because we trained with a custom form.
We can see that the first field, for example,
that we labeled this second signature, it's a string.
The valued string is married or extracted, and the text,
it's on the first page,
it's bounding box and it's confidence.
There's also a reference to
the extracted texts in the read results section,
if you want to look at the confidence
for the words that were extracted.
So this confidence is the key value pair match.
In the read results, you also have the confidence
that the OCR extracted this word correctly.
We can see also the other evaluate label is SSN2,
it's also a type string,
the value of the string,
the text, the bounding box, and the confidence.
Note that when we label,
you can assign different types to different values.
So you can assign a type string,
a type date, a type currency, a type number,
this will enable the value string to be
standardized and also improve the extraction quality.
In the Custom Train With Labels demo we saw you
can easily train a model to extract the values of interest.
Let's see how customers and partners are using Form Recognizer.
Automation Anywhere,
a leading robotic process automation provider has partner with
government health organization and other first responders
to streamline their operations and accelerate emergency response.
The National Health Service in the UK currently
processes tens of thousands of COVID-19 test daily,
and that number is expected to
increase to over 1,000 tests per day.
Before any sample can be processed,
an E28 form must be logged into the system.
The high volume of forms paired with the manual entry,
has the potential to cause
significant delays in testing and diagnosis.
Additionally, once a patient has
been confirmed positive for COVID-19,
several case report forms must
be filed with the World Health Organization.
Microsoft and Automation Anywhere work together to
automate the World Health Organization form
for reporting COVID-19,
rapidly making essential data available in
a digital form for
the National Health Service in the United Kingdom.
To help accelerate the reporting and
processing of many complex forms,
including the CRFs and E28s.
This highly secure solution is
comprised of Automation Anywhere RPA,
with native intelligent document processing
and Microsoft OCR and Form Recognizer.
The solution automatically extracts
and processes data from forms,
including handwriting and checkboxes.
It can be rapidly deployed on Cloud or
on-prem and can scale to
handle tens of thousands of pages of data.
The solution enables to easily digitize the information to improve
decision making and offload frontline staff
to spend more time on higher-priority tasks.
We saw a Form Recognizer can extract text and tables using layout,
train a custom model tailored to your forms.
Form Recognizer also includes prebuilt models,
where the first prebuilt we released is
a receipt prebuilt for US Sales Receipt,
like store receipt, and restaurant receipts.
The prebuilt receipt models is an off-the-shelf model
which has only one step, analyze receipts.
You can send your receipt using the analyze receipt
API and it will extract key fields in those receipts,
like merchant, subtotal, total, tax,
line item from English Sales Receipts in the US.
The prebuilt receipt extracts data from US Sales Receipt
like restaurants receipt and store receipts.
It will extract the following fields;
merchant name, merchant address,
phone number, date, time,
subtotal, tax, total, and tip.
It will also provide the receipt type
if it's it itemized, credit card,
restaurant receipt, the field
confidence level and also the line item,
the quantity, the product, and the amount.
While this data will be automatically extracted once you send
a receipt to the Form Recognizer analyze receipt model.
In this session, we saw a Form Recognizer can extract text,
tables, and key-value pairs.
Now customers integrated are now using
Form Recognizer following our additional resources on
Form Recognizer. Thank you.
[MUSIC]