Hello everyone!
Welcome to the presentation of our work about deep fake eyes and gazes,
in other words, Where do Deep Fakes Look?
Synthetic Face Detection via Gaze Tracking.
I am Umur Aybars Ciftci, and this work is a collaboration with Dr. Demir.
Since the conception of generative adversarial networks in 2014, their most believable results
emerged in face domain, leading to deep fakes.
Deep fakes are synthetic portrait videos of
real humans, photorealistic enough to fool humans.
Deep fakes are heavily exploited in
domains like media, advertisement, and entertainment, as positive use cases.
Deep fakes succeeded
in not only creating non-existing people, but also changing the age, pose, and race
as seen on the bottom right, or swapping faces of us to Jason Mamoa and Nicolas Cage, as
seen on the top right.
Following up, advancements in processing power and availability for generative approaches
have reduced the required resources and skill level for realistic image manipulation, precipitating
…
…the rise of the deep fake era.
Although this technology has been developed with positive
intent, it also attracted malicious users to exploit deep fakes for political misinformation,
impersonation, forgery, and pornography.
This threat to information integrity has consequences
in privacy, law, politics, security, and policy, thus, poses a threat to form social erosion
of trust.
As a defense mechanism, researchers introduced several deep fake detection methods.
These
methods can be categorized in two main branches as image domain and biological domain, while
the first one utilizes saturation cues, compression artifacts, and color consistencies; whereas
the second one exploits inconsistent head poses, blinks, face warping artifacts, blood
flow, and biological signals.
As the eyes are the mirror to the soul, we investigate the truth in the eyes of deep
fakes.
Comparing real eyes to the fake ones, we observe that there are significant visual
and spatial artifacts as shown here with four different generative approaches.
These artifacts
happen both between frames, and between left and right eyes; also causing broken symmetries.
In addition, anomalies occurs in fake gaze behavior.
Normal human gazes either converge
in far or at distant (excluding some irregular eye conditions), however fake gazes show divergent
or are completely off-plane behavior.
Moreover, even if the direction is corrected, the continuum
of 3D vergence points is broken, as fake gazes exhibit noise, missing paths, and demonstrate
non-uniform distribution.
In addition to the analysis of previously mentioned features, we also evaluate their
properties in temporal domain for fake and real videos.
For example, real gazes cover
more vertical and horizontal range than fake ones, on the other hand, fake gazes have shaking-like
noise.
The fake iris sizes can differ by 30mm2 and pupil sizes by 14mm2 for the same person,
whereas real eye sizes are almost constant.
Such signal properties are also detectable
in the spectral domain.
We formulate all these geometric, visual, metric, temporal, and spectral features to
gather all eye and gaze related consistency and coherence measures together.
Visual features
include area and color of eye, iris, and pupil.
Geometric features include gaze vectors, vergence
approximation and error, interpupillary distance, and misalignment.
Metric features is obtained
by the cross correlation of visual and geometric features, and temporal features cover visual
and geometric features from all frames of a sequence.
Finally, spectral features are
the power spectrum of the temporal features.
Then, we create gaze signatures with these features from different domains, of size 40
by omega by 3, as listed on the right.
Looking closely to the signatures, some significant
differences of real and fake videos are visible, for example the power spectral density of
areas of different parts of the eye through frames is much colorful for fake eyes, or
cross correlation of left and right gaze vectors are much uniform for the real ones.
We feed the gaze signatures into a simple three layer network, documented on the right.
We have the comfort of training a simple network due to the powerful representation we generate
with gaze signatures.
We train and test on equal number of real and fake sequences, with
a random 70/30 train/test split.
We refer you to our paper for cross validation experiments
with different splits.
Our eye/gaze based fake detection algorithm achieves 99.27% accuracy on DeeperForensics
dataset and 92.48% accuracy on FaceForensics++ dataset.
We note that if a deep fake generator
carves out the eyes, our approach behaves like a random guess, as reflected in our Face2Face
accuracy.
We compare our results to the detection results of eleven fake detectors, utilizing
either complex deep networks or other biological features, and we observe that our approach
overperforms almost all of them, except FakeCatcher.
We conduct ablation studies in order to evaluate the contribution of different feature spaces
on the accuracy.
We conclude that geometric features are more important, and metric features
contribute less.
We also experiment with different sequence lengths, finding out as omega=32
as the ideal value.
Finally, we measure our robustness against postprocessing operations.
We discover that features from different spaces balance each other, i.e., high resolution
eyes enable geometric features and low resolution eyes enable visual features.
On the bright
side our approach is 55 times more robust than fakecatcher in this context.
Our detection
only drops by 0.37% with 9x9 gaussian blur, whereas FakeCatcher drops more than 20%.
Finally, we compare our approach and FakeCatcher, observing that the failure cases are mostly
mutually exclusive.
We correctly classify a fake video on the left, and a real video
on the right, whereas FakeCatcher fails in both, due to physical or digital corruption
on the skin.
We propose ensemble detection platforms where such detection algorithms
can be cascaded for complementing each other.
To sum up, we demonstrate that humans contain natural watermarks that can be exploited for
deep fake detection.
Eye and gaze features are of such signals, and formulated carefully,
we can achieve high detection accuracies on several datasets.
For future, we would like
to witness fake detection platforms combining multiple biological priors in ensemble algorithms.
Thanks for listening our talk.
You can direct your questions right now, or you can also
reach us from the information on the screen.