Welcome back to the OpenVINO channel AI enables computers to understand what we
see, what we talk, and now even what we write..
Even if you are a mathematician..
This nice python demo can detect free form handwritten formulas or ones written by Latex
Google OpenVINO formula to get to the full documentation
This demo is done by a model divide into 2 separate models..
The first model, called the encoder is a convolutional neural network used to extract features out
of the image.
Basically, the first stage of identifying the bounding boxes of the letters or symbols..
The second model is an LSTM model Because the context is important, we need
a model that can remember the previous symbols..
And understand the full sequence So until it reaches the maximum formula length
that you can define in the command line, it will look for the next symbol,
Based on the inputs provided by the encoder model.
Overall this demo has many options and parameters, look for the documentation
Each of the models can do slightly different things
For example, the Latex model can detect upper- and lower-case English and some Greek letters,
numbers, functions and more
The handwritten model can predict letter numbers written by hand..
First, let’s define the OpenVINO directory and my working directory
Initialize OpenVINO And go to my working directory
And let’s look at the models we need.
Under the formula recognition directory you have the list of models required
You can see here 4 models.
2 for Latex, decoder and encoder And 2 for handwriting formulas
Let’s get these files, Using the downloader utility
Please bring me all the files on this list To my working directory
And I need on FP16
And the 4 models are here
Now I need to look for the vocabulary files..
Here are the two of them..
I’ll just copy them both to my directory And I’m all set
What I have in the directory is two images of Latex and free handwritten forumlas
the vocabulary json file and the models..
Let’s run the sample..
Formula recognition python This is my encoder model.
“Im2latex “ encoder And the decoder model
Vocabulary file And my input image … which looks like that
And it’s running, You can see 474 W to the power of 1 and so
on..
So this is working well
For the free handwritten formula I’m using the same sample code..
But the hand written encode model And decoder
On my input image , that looks like that..
And it’s running
51d + 0.6 … looking good..
If you are using the 2021.3 OpenVINO version, you can use the interactive mode
Which is really nice..
Here is how the Latex demo works..
Using the Latex JSON file, the input will be a web cam
And I made the recognition threshold a little lower and the resolution smaller
It takes few seconds, but you can see that the Latex formula is being detected..
The handwritten one less, so let’s try the other model
And this is the handwriting models with the same parameters
And the handwritten formula is well detected
Very nice demo and another usage by OpenVINO You have here QR codes to many of the resources
OpenVINO offers, Feel free to connect with Ray our evangelist
And don’t forget to subscribe to the channel to get more videos like this one
Thank you