We will now have Ke Liu from the Shanghai Jiao
Tong University to present smartphone-based handwritten signature
The x-axis represents time and the y axis represents taps, which
verification
using acoustic signals, please share your screen and (okay).
In when you're ready.
Thanks for your introduction.
And hello, everyone, I'm Ke Liu.
It's my honor to present our work, the smartphone-based
handwritten signature verification using acoustic signals.
The handwritten signature is one of the oldest biometric
authentication for paper-based documents, which can represent
the identity of a person
and the intention of that person has been commonly accepted and widely
used in business, financial and legal areas. Nowadays,
signature methods can be divided into offline methods and online
methods, the offline
methods against the static images for verification
while the online methods using a special device to capture
dynamic handwriting patterns when signing, existing online methods
usually touch or wear, either a special signature device,
such as a sensor-instrumented pen, a tablet, a glove or a
smartwatch.
However, this additional special devices restrict the practicability
of these online approaches.
Smartphones with various built-in sensors have become indispensable
in our lives, which open up new possibilities for online signature
verification.
Although there are already some existing work that leverage inaudible,
acoustic signals to capture action patterns for activity recognition,
automatic handwriting signature verification
using acoustic signals has not been fully explored. To achieve a
signature verification, a of phase.
method is to leverage a smartphone to transmit inaudible
acoustic signals, which means the frequency range
is about 18 to 22 kHZ and record echo signals, reflected by handwritten
actions to capture handwritten, behavioral characteristics, and
physical characteristics.
And before describing our handwriting signature verification system, I
want to envision an application scenario at first to begin ways.
It can verify signatures automatically and immediately and when
performing online signature verification, users don't need to carry
some unusual devices, but only smartphones.
And in addition, it can serve as our assistant or additional
handwriting signature verification
approach. In a two factor authentication system,
however, there are some challenges to fulfill this system.
The first is the hand motion of signature is so tiny and we use CIR
that is Channel Impulse Response to capture these features.
And we design our special up-sampled GSM sequence to expands the
signals' sensing range.
After getting the signature motion features,
we proposed an attentional, multimodal, Siamese network to do a
verification, the network, the network can achieve, a good performance
by emphasizing the interpersonal, variability, and accommodating
interpersonal
variability.
And in the second part, I will talk about some details about our
system, SonarSign
Here is an overview about SonarSign
It consists of four major components including two signal
collection modules, acoustic signals, transceiver and CIR
estimation as well as one CIR
information processing module and one deep learning module
signature verification.
And the the pipeline of system is at first, the speak on the
smartphone, sent out the signals and the microphone receives that echo
signals.
After receiving the echo signals, CIR estimation is performed
and multimodal information is extracted.
Then it will be fed into the network mode to perform a verification.
The key of the first module is that design of signals.
The baseband we select is GSM sequence.
It has 26 feet and weep at 14 B, 0, symbols in its tail to prevent
frame corruption.
Then we up-sampled the baseband our attempts to expand the
sensing range and ensure the range
resolution, the sensing range increased to 42.85
centimeters, and the
range resolution is 0.354 cm then we modulate the signal to
inaudible frequency range. After that,
the signal can be placed through speaker
And the microphone in smartphone can record the echo signals
After recording the echo signals.
SonarSign downsampled echo signals to the baseband with standard I/Q.
demodulation, then performs CIR estimation.
The figure in the right is the visualization of CIR.
represents the propagation path of
signals.
Here we do
an experiment to demonstrate the feasibility of using CIR to extract
dynamic motion features.
Two participants are asked to rate the same signature many times.
and the result shows, dCIR, from the same participants, for
the same signature.
inhibits similar patterns, which demonstrates the stability and
repeatability of the handwritten signatures.
while dCIR
are forged signatures shows apparently different patterns to
genuineness, and this discrepancy is large enough to distinguish them
from each other.
And the last part is the deep learning model.
We proposed our attentional multimodal CMEs Network.
The input of this network is a pair including evaluation sample, and
the enrollment example. For each sample, three modalities are extracted.
Before the network has two branches, which we should selling ways with
each other to perform the same feature.
Course as I Revenge has three CNN subnets which is so as
feature extractor for three different modalities, after extracting the
otherwise features from input sample and attentional module is
followed to align the two feature embeddings. Finally, four
connection layer is followed to compute the results.
By this state, by this pipeline, we got the final results.
And the third part
I will talk about some details about evaluation.
The first is the information of datasets.
We collect two data sets.
One is basic dataset and the other is unseen user data set.
For basic dataset, it contains 23 participants and for each
participant, he or she has 40 positive samples written by him or
herself.
And 35 skilled forge samples, imitated by others, and for
unseen user data sets,
It includes 12 participants, which are not shown in basic datasets.
And each one has 20 genuine signatures and eight skilled forged
signatures.
We use AUC and the EER to evaluate the performance of the system, the
good handwriting, signature verification system will have a high
AUC with a low EER value and the picture below shows the same when
collecting this.
And first, we perform 5-fold cross validation on basic
datasets.
And we got AUC equals to about 99% and the, EER, is about 3.27
percent on average.
These results manifest that SonarSign kind of effectively
distinguish the legitimate signatures from false ones.
Also we test it on unseen user data sets and we got AUC about
98% and the EER, is 5.79%.
It demonstrates
Our system can extract generalized features of handwritten
signatures, even for unseen participants.
We also compare our measures with some existing methods using acoustic
signals by testing on the same data set, the results
show our system has better performance.
Besides we test our system on different positions and orientations. The
results show it can still perform
Well within the appropriate variation range.
And some environmental interference is also considered. In practical
scenarios,
there may be a pass-by or music playing, when someone signs. To
superpowers influence from this environment factors may affect the
system performance, this this experimental results also show the
robustness of our system.
then we do some micro-benchmark tests, including different
network models, different modalities and number of participants in
training sets to demonstrate the effectiveness about our
system.
Okay, in conclusion, we elaborate an automatic handwriting
signature verification system, using handwriting signature patterns,
captured by inaudible acoustic signals with commercial off-the-shelf
smartphones and its extensive experimental results envision
that SonarSign can conveniently correctly and robustly
verify signatures.
However, there are still some works waiting to be explored, such as
compatibility on different smartphones accuracy for signatures with a
longer time span.
and the combination with other verification approaches.
This, there are works that need to be done in the future.
Okay, that's all.
Thank you.
Thank you very much for your talk other.
It looks like a really nice system that can come in handy while often.
Are there
any immediate questions in the audience or in the chat?
Okay, then I would like to ask a question.
Did you consider different languages?
Because there are some languages that are written from left to, right?
Right to left top to down.
Some are one fluent
Motion other are different single strokes.
So did you consider that if that will have an impact or?
Okay.
Oh, I know.
I know y'all.
I mean, and the scenario we can say there is the normal
thing, which means your sign your signature seeing a normal way to
like, you're just like what you doing the laugh, or there's no, some.
Yeah, sorry.
There are no some strange situations here.
Okay, so it should your mobile should work
Independent of the language.
the person is signing with? yeah our people mentioned the the signing
in Chinese as
collect the signatures are written in English and it's
shown the performance.
The EER
The ER value has growth.
and you know, the lower, the value of EER, the lower the better,
but the value increase,
If you use some transferring measures, which means your offer,
offer
some samples written in Chinese and the results will be better.
Okay.
Thank you there.
Any questions from the audience?
Well, if not, I'm going to ask a second one.
What are the implications for your system?
So, are you actually planning on having this system installed on
personal smartphones?
So if you sign a contract with someone, you get out your smartphone
and basically check that it's basically them signing or what are
basically your implications for the future.
Sure, how will the system be be used?
as a yeah, I, the way to suppose the scenarios is as when people need to
send important documents and he or she and put the smartphone aside
and then sign the smartphone will upload the recorded audio
information to our qualified company or institutions for
authentication, which means the influence of the network model is
not executive in
In the following impractical scenarios.
It will be run in the server of qualified company or Institutions.
This ensures fairness and security.
Okay, that also takes care of.
another question
I had, how to distribute these patterns, but if it's centralized and
takes care of it.
Thank you.
Okay.
Thank you.
So, the question asks.
Thank you for the presentation.
I just wanted to ask the moment, you use the CNN.
Did you consider the false positive for your - musical education?
Sorry, can you speak a bit louder?
Yeah, I can repeat the question.
So when you basically with the CNN, did you consider the false
negative in any form?
So what do you mean?
False negatives?
It's the negative samples.
That was forged was recognized as a correct signature.
Yeah.
Do you mean
what signature can be regarded as a false signature?
Yeah, I, the false signature is the signature imitated by the emitters,
which means it is not stand by the correct user.
Yes, but the system recognize one of these signatures falsely as the
correct signature.
I think that was the the question.
yeah, yeah, it means
the same, maybe occur, you know, the AI system can't access, the
100% accuracy.
This is not, uh this, it is.
It is must be happened and we can't prevent it.
Okay, does that answer your question?
Thank you.
Okay, and then we're out of time.
So we would go through this to the third talk.
Thank you very much again.