Hello,
I am Ishani Janveja and I’m here to present InSight, a system for monitoring the State
of the Driver in Low-Light using Smartphones.
Road Safety is a major public health issue across the globe with about Two thirds of
the accidents occurring during night-time or in low-light conditions
And Driver fatigue and distraction are the leading causes in such scenarios.
We believe that the way to address this issue is by monitoring driver behaviour and providing
actionable feedback.
In this work we focus on developing a vision based system such as the one shown in the
figure - where we simply mount a smartphone in the vehicle for monitoring driver’s actions.
However, monitoring in low light conditions using standard smartphones poses several challenges
The first one is that RGB frames captured by the smartphone lack information.
For instance, in the image on the right, we can hardly see anything, let alone monitoring
the driver’s behaviour.
The solution for this is to use IR or FLIR cameras.
But, this requires additional camera support.
Thus making it an expensive and intrusive solution, and hence not scalable.
The second challenge is that pre-trained models for face and landmark detection do not work
on low-light images simply because they are trained only on images taken in good lighting.
For example, using a pre-trained model we were able to detect faces in only 1% of 1000
low-light images from a real-world drive.
So how do we counter these issues?
In our solution, we retrofit existing smartphones with a near-infrared LED, as shown in the
image, to capture driver’s actions in low light.
Unlike existing solutions, we repurpose the smartphone camera to capture NIR imagery,
making the system low-cost and also non-intrusive
So can we use smartphones to capture NIR imagery?
The short answer is YES.
Typically, Cameras are manufactured by placing IR filters in front of the light sensors,
thus blocking the infrared spectrum shown in the grey region.
However, in case of smartphones, which are mass-manufactured and often the aim is to
push costs low, the IR filters do not completely block out the near-infrared spectrum as shown
by the pink region.
This provides us with an opportunity to use smartphones to “see” in NIR light.
Now, what kind of NIR LEDs can be used effectively with smartphones?
As you see in the images captured by the smartphone, when we use LEDs with wavelength close to
that of visible spectrum, more red color is visible making the system intrusive.
However, the filter blocks out most of the light on the other end of NIR spectrum,.
Thus we choose the 810nm LED which provides sufficient illumination.
Another point of consideration is the intensity of the NIR LED which is proportional to the
power consumption.
We used LEDs with 1.8W to illuminate the driver’s face which is typically at a distance of 40-50cm.
Now that we have NIR images captured using smartphone, our goal is to
First, detect the driver’s face and localise facial landmarks
Then use it to monitor driver fatigue based on eye blinks and yawns
And lastly, to measure attentiveness based on driver’s gaze
In order to train our facial models, we first need to generate labelled data.
However, labelling 68 facial landmarks manually on all training images would be laborious
and also infeasible.
Since IR imagery is much more effective in low-light and can be used to extract labels,
we leverage this insight and capture IR and NIR images in tandem using the setup shown
on left
We then translate labels from IR images to images captured by the smartphone in NIR illumination.
This is done using Homography Transformation.
To demonstrate this process: We have an IR image
which needs to be processed to remove the radial distortion due to the fisheye camera
lens.
We then detect landmarks on this image using standard models
and finally the Homography Transformation Matrix lets us map the landmarks to the corresponding
NIR image captured using smartphone.
We use this labelled data to train our models.
To evaluate our system, we collected a dataset of driving videos across 15 drivers.
Some of it in the basement and rest on road for real-world scenarios.
Our face detector trained on NIR images gives an accuracy of 90% and the landmark predictor
gives a Normalised Root Mean Square error of less than
0.08 which is well within the acceptable error.
Next, we have the results for blink detection and yawn detection on the basement data with
150 blink events and 180 yawn events.
The F1 score for detection of both these events is around 85%, showcasing the effective
ness of our NIR imagery based models.
We also demonstrate here our evaluation on real-road data for mirror scanning.
In this graph you see the percentage of time when a driver scans the left or right car mirror
at least once in a window of “T” seconds.
We see a general trend of decrease in adherence to this rule from evening time to night time.
To summarise,
We presented InSight - an accurate, low-cost and non-intrusive system to continuously monitor
the state of the driver in low-light.
Our system relies on NIR illumination which is not visible to the human eyes, but can
be captured effectively by smartphones
We conduct a rigorous evaluation of the system in different settings.
Further, we also present an alternative approach based on synthesising thermal images using
GANs for monitoring in low-light
For more details, please read our paper.
Thank you!