(upbeat music)
- Welcome back to Azure AI Essentials.
Today we're going to look at how you can add
optical character recognition
and AI-powered search capabilities to your applications.
We're going to focus on document processing,
a popular scenario,
but the concepts you'll learn here
will apply to other scenarios as well.
Organizations today are processing
more documents than ever before.
Think about paperwork needed to buy a car,
filling out tax forms,
or documenting work you've completed.
Processing a large amount of documents is a reality
for every organization and industry.
Now think about how much time it takes
to process all those forms,
and the variety of forms that need processing.
Internal forms may have to be handled differently
than customer forms.
And if they're filled out by hand,
it may be hard to read all the different types
of handwriting.
Processing all these documents manually is time consuming,
inefficient, and leaves a lot of room for human error.
Organizations across the globe are integrating AI solutions
into their workflows to help automate document processing.
Let's get started by walking through a typical
document processing scenario.
This diagram represents main components
for automated document processing built on Azure AI.
Over the next few minutes,
I'll walk you through these components,
OCR, Form Recognizer, and Cognitive Search.
We'll start with optical character recognition or OCR.
Handwritten text can be difficult to parse through.
With Azure Cognitive Services,
you can leverage OCR capabilities
that can extract both printed and handwritten text
from images and documents.
You can add OCR capabilities to your app
using a portal experience,
or with Computer Vision APIs or SDKs
for C#, Java, JavaScript, or Python.
The Computer Vision Read API
extracts printed text in several languages
and handwritten text in English
along with digits, currency symbols and more.
It's optimized for text-heavy images
and multi-page PDFs with mixed languages,
and even supports detecting printed
and handwritten text in the same image or document.
The first step is to use the API's Read call
which takes images and documents as input,
and supports a variety of file formats.
For large multi-page documents,
you can provide a specific page number or page range
as an input parameter to extract text just from those pages.
Here we have a variety of document types.
As the Read API parses through each of these documents,
it uses OCR to detect the text and read what's in them.
This also works for documents
that have mix of languages on the page.
You can see that the computer vision model
detects each language
and is not confused by the images on the pages.
In each of these samples,
the API is called and extracts text
from the image or PDF input,
and returns a JSON response.
For each line of extracted text,
the JSON will output the line of text
along with bounding box coordinates and a confidence score.
As you just saw,
OCR can be really helpful to reduce the time it takes
to process images and documents.
The next Azure AI service that will help you
streamline document processing is Form Recognizer.
Form Recognizer can accurately extract text,
key value pairs, and tables from documents,
so you can quickly turn forms into usable data,
with no machine learning expertise required.
Similar to OCR,
Form Recognizer is built on top of the
Computer Vision Read API.
While OCR can detect text lines and words,
Form Recognizer can understand document structures,
fields and values.
Form Recognizer is comprised of three parts,
the layout API, pre-built models, and custom models.
Let's take a brief look at each one of these.
We'll start with the Layout API
which can extract text, tables, and check boxes
from documents.
Given a form, Form Recognizer detects each cell in a table,
as well as numerical values, currency symbols,
and characters.
In the JSON output,
the rows and columns are indexed,
along with their bounding boxes and elements.
Next, we have pre-built models.
Form Recognizer currently supports two prebuilt models,
sales receipts and business cards.
The business card prebuilt model extracts information like
name, title, address, and company.
The receipt model is optimized for
common English sales receipt formats
like those used by most restaurants, gas stations
and retail stores.
The model extracts key information such as
transaction time and date, merchant information,
and tax amount and returns it in a JSON file.
Finally, we have custom Form Recognizer models.
You can train custom models with your data,
to tailor the model to forms specific to your business.
For example, this W2 form does not follow a standard table
of rows and columns,
but has more of a key value layout
with different boxes and fields.
So it's best to use a custom model here
rather than the Layout API.
Training Form Recognizer custom models
does not require manual data labeling,
and you can get started with as few as
five sample input forms.
If your form images are of lower quality,
we recommend using a larger data set such as
10 to 15 images to train your model.
This way your model has a better chance
of extracting information.
Azure also offers the Form Recognizer labeling tool
to select fields from your form
and train your model in a supervised approach.
This is especially useful if Form Recognizer
is not picking up all the content you're targeting.
First, you'll create a new model,
then choose a custom image or form to analyze.
You can train custom models with labeled or unlabeled data.
Form Recognizer uses unsupervised learning
to understand the layout and relationships between
fields and entries in your forms.
When you submit your input forms,
the algorithm clusters the forms by type,
discovers what keys and tables are present,
and associates values to keys and entries to tables.
You can see here that this custom model
detected the important information in this form,
the check boxes, severity, and more.
Now let's look at the JSON which returned the read results,
page results, and document results.
Right away you can see which model was used,
in this case a Unix one,
and even the submitter's name and bounding box.
Whether you're processing documents,
receipts, business cards,
or forms unique to your organization,
Form Recognizer can help extract valuable information
from them.
The final step of automated document processing
is searching your data.
Azure Cognitive Search is the only cloud service
with built-in AI capabilities
that enrich all types of information
to easily identify and explore relevant content at scale.
Cognitive Search can be broken down into three steps,
ingesting your data,
enriching your data,
and exploring your data.
We'll start with ingestion,
the first step to indexing your unstructured data.
You can create an Azure Cognitive Search resource
in the Azure Portal
and connect to a data source from there,
or you can do this programmatically.
You can ingest data from Azure or third-party sources,
across a range of formats including
images, audio files, PDF, CSV, and more.
Now that you've ingested your data,
the next step is enriching your data.
Data enrichment is the process of
enhancing your understanding of your data
with cognitive skills,
for example, image classification, or language detection.
Enriching your data allow you to produce new fields
in your search index that are not available natively
in the source.
You can leverage pre-built AI models
with Azure Cognitive Services,
or build something custom.
One thing to remember here is
to make sure you create your cognitive services resource
in the same region as your cognitive search resource.
Applying AI capabilities such as OCR,
translation,
key phrase extraction,
and entity detection
is as simple as checking a box
in the Cognitive Search portal.
Next, we're going to customize our search index
by defining the index properties.
The Cognitive Search resource
automatically identifies metadata
related to the files we chose earlier,
but you can customize by making each of these fields
retrievable, filterable, sortable and searchable.
So now you've ingested your data,
added cognitive skills,
and created an indexer,
but what does this mean for our
document processing scenario?
So far, we've been setting up our search criteria
so we can easily surface valuable information
rather than going through it manually.
You can select how often your data is searched and indexed.
If you're ingesting documents every few minutes,
like at a doctor's office when patients are checking in,
you might wanna index your data hourly.
If you're expecting field technicians
to send in daily reports,
you can index your data daily.
Finally, you're ready to explore your data.
You can integrate custom models or classifiers
that can be built with the language
or framework of your choice
and explore insights with pre-configured search
or through analytics tools and business applications.
In our example here we can search for the name of
Microsoft's CEO, Satya,
and as you can see we get a letter that he wrote.
All the other fields we'd previously defined,
people, organizations, locations,
these are also extracted from these documents.
Let's try one more, let's search bot.
Now this text is surfaced in Spanish,
but because we chose for our indexer to translate it for us,
we can scroll down and see the same information
extracted in English.
Cognitive Search works with several file formats
and built-in indexers,
and supports over 50 languages.
You can query your data through APIs or the portal,
to quickly transform raw, unstructured information
into searchable content
to make your document processing workflow more efficient.
This is just one example
of how organizations can use Azure AI
to modernize their business.
There are many ways you could extend this solution
by integrating with other Azure AI services.
For example, you could use Personalizer
to provide personalized content or document recommendations,
or Metrics Advisor to detect odd behavior
in your company's data.
And you can apply OCR, Form Recognizer, and Cognitive Search
to help solve a wide range of business challenges
beyond document processing.
Thanks for tuning in to Azure AI Essentials.
(upbeat music)