Tagging complex documents can be a huge hassle, especially
for our mobile users.
Today on Developer deep dive, we'll be talking about some
of the machine learning algorithms
behind our auto-tagging technology.
[MUSIC PLAYING]
Hi, my name is Matthew Roknich, a developer advocate
at DocuSign.
Joining me today is Raphael Alabi, PhD and senior machine
learning engineer.
Raph, it is great to have you on the program.
Thank you so much for having me, Matt.
I really appreciate it.
So before we begin, I want to ensure that all of our viewers
understand some key DocuSign terminology and the most
basic e-signature object model.
So the DocuSign object model consists
of 4 kinds of objects, envelopes, documents,
recipients, and tabs.
Tabs and fields are locations in a document where
a recipient must provide input.
This could be a signature, text, initial,
or any number of other types.
So, Raph, where does this flow go awry?
Thank you.
I think what we noticed was that our mobile application
users were having a hard time processing fields
on their documents.
The pain point got exponentially more frustrating
when there were many fields to be positioned.
For example, if you take a look at the document to your right.
What you find out is that that document is very, very easy
for anybody to tag without any issue.
However, as it goes towards your right, sorry that was left,
as it goes towards the right, you
find out that the documents get progressively more
difficult to tag, and even probably impossible for a human
to tag.
Well, that's where we decided to collaborate with Google.
And we came up with this architecture,
which is the auto-tagging architecture or algorithm,
which can tag documents, irrespective
of their complexity, in around 2 seconds.
To enable this technology, what we had to do
was marry 2 areas of machine learning, Computer Vision
and Natural Language Processing.
Now computer vision is a branch of machine learning
that tends to solve category of problems
like classification, localization, object detection,
et cetera.
Natural language processing, on the other hand,
tries to solve an array of problems
where semantic and syntactic recognition of language
phenomenon are vital.
Problems that are solved by natural language processing
today include machine translation,
information retrieval, sentimentalizes, et cetera.
By marrying these 2 technologies together,
we were able to successfully deliver
an auto-tagging technology that today caters to and provides
a better experience for DocuSign customers.
Now the computer vision problem that we solved--
Basically, you can think of that as something
that is easy to solve.
And we solved that using basically third party
algorithms that are available in the market.
However, the natural language procedures stuff
is what we found to be much more interesting.
Now the architecture that we eventually came up with
is shown here.
Whereby you basically have a PDF document
that you get from either from an Edge device,
from a mobile device, or from the web.
The PDF document gets converted to images,
and these images get fed into an OCR, which is an Optical
Character Recognition tool.
And the OCR, its function is basically
to extract the tokens, then after the tokens have
been extracted, this is passed through
a recurrent neural network.
And eventually we're able to accurately tag labels
on every page from the PDF document.
To go into more detail, so we take our PDF documents.
And the PDF documents, they are converted into images.
And the images, they can be of different Dots Per Inch,
or DPI.
And as we go higher in DPI, we actually
get much more larger files, and it also
becomes a problem, transmitting this across the network.
So that's why we eventually settled with 76 DPI.
There were some loss in accuracy,
but eventually we did get accuracy of up to 95%,
which is really good for the performance
and for the problem statement that we had at hand.
So after we've taken our PDF documents,
have converted into images, the next step
is now to pass it through an Optical Character Recognition
tool.
Now the OCR that we use, because we work in collaboration
with Google, we use the Py-tesseract, which
is a publicly available OCR engine that
was developed by Google, or that's
been maintained by Google.
And eventually what that does is that within 200 to 500
milliseconds you can actually get the tokens, which
are the words on the document, as well
as their respective locations.
And this is where we actually feed into the next steps, which
is on the next page.
So when we get the tokens, or the words and their locations,
our next goal is eventually to be
able to use those words, their locations within the document,
to eventually characterize a Bounding-box.
And we use multiple approaches.
One of the approaches being basically just
to select an arbitrary static Bounding-box.
When we did that, what we found out
was that because there's no rule in any language,
and much less so in the English language,
as to where a particular token or word that [INAUDIBLE]
Bounding-box should be.
You can find situations whereby you don't actually
get to select the one that is correct
or that correctly identifies or characterizes that Bounding-box
that you want to tag.
So because of that, we shelfed this,
and then we went to another approach, which is instead
of us actually using a physical Bounding-box, what
we decided to use was just to have a varying Bounding-box.
Basically, what I'm trying to do is
to look for locations outside of your Bounding-box
until you actually either find a word,
or you find another Bounding-box.
And when you do that, you extract the word.
And you take the word as a group of words that eventually should
characterize that Bounding-box.
And after this is now done, we need
to be able to eventually place a location to the Bounding-boxes.
Just to clarify, what you're trying to accomplish here
is determine which token applies most accurately
to the Bounding-box.
Is that correct?
Exactly.
Correct, yes.
Great.
So how do we accomplish that?
How do we figure out which one is most applicable?
Very good question.
So eventually what we settled with was
we use basically the rule of the English language.
Or maybe the accepted rule of English language whereby
if you're trying to--
Normally, English is written from left-to-right, right.
So because of that, if you're trying to actually characterize
a Bounding-box, more than likely you're
going to characterize the Bounding-box using
the left token, the token to the left of the Bounding-box.
However we've seen situations where like,
token's on the right.
But more than likely, it's going to be from the left.
But then you can also have situations whereby
there's no space on the left.
And then the user eventually has the attributes
of the Bounding-box occurring either at the top
or at the bottom of the Bounding-box.
And this happens, more importantly,
for cases like signature.
As you probably see that the signature's probably
going to be at the bottom of the Bounding-box, and then
you now see it.
So eventually, what we decided to do
was just take the location of the Bounding-box
into perspective and apply a way to take you to those locations
before we take those words and then pass those words
into a recurrent neural network.
When I said we take those words and pass them
through a recurrent neural--
I lied a bit, because words are not
what you're passing through a recurrent neural network.
You actually need to pass vectors,
they take floating-point numbers.
And so what I'm showing here is basically a way,
whereby, we converted those words into vectors.
So just showing here is just a mapping.
Let's just say it's an example, here.
Essentially we're trying to map the words like king, queen,
man, woman, smart, et cetera, into a hundred or a 300
dimensional vector space, whereby each specific vector
actually has a representation.
What we have here is royalty, gender, and then,
we also have intelligence.
So for like the king, we expect that a king is a royal person.
If you hear the word king, it's probably like 100%
that person is royal.
And then, gender it's probably that, maybe it's
a male or a female, most likely it's a male.
And then, intelligence is we expect
some level of intelligence from a leader, right.
So those are what vectors here that characterize a king.
And then the queen will be probably
something similar but of a negative gender,
because a queen always characterizes a female gender.
So you have the same kind of attributes,
and you have your intelligence.
Your intelligence will be all 0.9
for intelligence vector space.
Both probably neutral, zero.
And zero for either gender or for royalty because, I mean--
[INTERPOSING VOICES]
Raph, where are we deriving these correlations?
Yeah.
These vectors that, we derive them from--
Google already has all of this done, and what we do
is just use these word vectors.
So we can either have, we actually explore three,
we can have [? Fac-X, ?] which is from Facebook,
which is a [? WaterVec ?] model from Facebook,
or you can have the [? WaterVec ?] itself,
which is from Google, or you can have [? GuaVector, ?] which is
from Stanford.
So we explored using all three, and eventually we
settled with using [? GuaVectors ?] for converting
our words into vectors.
Got it.
And after the words that we converted to vectors, then
the next step is passing these vectorized words
or the vectorized version of these words into an RNN.
So when we pass it through an RNN, the function of the RNN,
you can think of it as taking each word or each 300
dimensional third vector, and then doing
something similar to a single word value
decomposition on those word vectors,
making them much smaller.
And then, passing, you can think of the succinct meaning
of those words are long.
And eventually, mapping those out
into a probability distribution space.
Whereby we're able to eventually find out, OK, well,
because we've seen all of these words that surround
a specific token, guess what, this means that this word here
is probably a signature, or this Bounding-box is probably
a signature, or a date, or a date sign, or an address,
et cetera, which is what is being shown here.
Effectively, so you have the words going
and eventually come out to give you a probability distribution
space, which you can basically, you can think of it as well,
guess what, the highest probability
is right here, and so because of that, that's
what the output is most likely to be the attributes
to the Bounding-box.
And so, this is most of it.
This is actually all of the architecture that we have.
And so, I hope I've been able to pass across to you
the very interesting technology that we
work with in collaboration with Google.
Whereby we are able to take PDF documents,
map them, compare them to images,
pass them through an OCR, and eventually pass them
through an RNN to get very accurate tags on Bounding-box.
That is used for documents and is
being used to eventually make life easier for mobile
as well as web users of the DocuSign ecosystem.
It's fantastic.
I can definitely speak for our DEB community
and say that I didn't realize how complex auto-tagging would
be.
And so, it's incredible to see how exactly it works.
Do you have a demonstration that you
could show us of auto-tagging manifested in one
of our products?
Oh yeah, definitely.
So as shown here, a document is being uploaded.
And you can see that's a team timeform PDF document.
And you just need to provide a name,
here we have a random name that's just been
provided, and email address.
And once that is done, you basically
output that to the DocuSign environment.
And then you get the option to either accept other tagging,
accept all the tagging feature.
And if you click Yes on that, what happens
is that you get your PDF document converted
to a PND file and then, sent to our auto-tagging algorithm.
And then eventually what you get back
is [INAUDIBLE] tag of documents.
And you can see this is really fast.
This probably ran for less than a second.
Wow.
Having spoken to a ton of developers at conferences
and hackathons, I can say confidently
that this is going to change the game.
Tagging documents, especially very, very complicated forms,
is not always fun.
And so, I'm very excited to see this used on a wider scale
by our customers.
So for the developers out there, if you're
interested to explore DocuSign's developer offerings,
please head over to the DocuSign developer center
where we have guides, API references, and tools
to help you get started with your integration.
And, Raph, it was absolutely a pleasure
to have you on the program.
Thank you so much.
Thank you for having me, true pleasure being here.
[MUSIC PLAYING]
If you like that video, be sure to subscribe
to our channel for the latest DocuSign developer content.
If you want to see a specific topic in a future video,
comment below.
For everything else, visit the DocuSign developer center.
[MUSIC PLAYING]