Hi, I'm Tong Wu from Rutgers and I'm Nik Martelaro from CMU. And we.
Are here to talk about our paper on learning when agents can talk to the drivers
using the Is now a good time to dataset and multi-sensor fusion.
This is work with our collaborators Simon Stent from the Toyota Research
Institute, Jorge Ortiz from Rutgers, and Wendy Ju from Cornell Tech.
In the future, proactive speech agents could be very useful for drivers.
However,
picking the right moment for the speech agent to interact with the driver can be
challenging due to the complexity of the driving environment and the fact that
interrupting a driver at the wrong moment could be distracting or annoying.
Here's an example of a driver being interrupted and not such a great time.
"Is now a good time".
Since we can capture various forms of data such as video, car telemetry,
and human physiology.
We've worked to build a multi-sensor fusion model that could predict good and
bad moments to interact with the drivers based on their preferences and
third-party ratings of safety. Using our first channel,
can we were able to achieve,
was there a 0.874 F1-score predicting good and bad moments to interact and a
0.891 F1-score score to predict safe and unsafe moments.
To train our model,
we use a subset of 1,915 samples from previously collected data
where drivers were randomly asked is now a good time on a 25 kilometer driving
route.
They then self annotated their preferences for interaction by saying yes or no.
Each sample includes in-cabin video of a driver with body pose estimation,
road video with traffic object detection, car telemetry,
and physiological data from the driver.
Because prior research has shown that drivers might be overconfident in their
decisions when to interact with car interfaces,
we also augmented the dataset by collecting readings of how safe a moment was
I asking third-party raters on Amazon Mechanical Turk.
Four Turkers rated the moment for how safe it was to interact with the driver.
On a five-point scale,
we then average these ratings and use those to train a safety model in addition
to training a preference model.
Since the dataset contains 22 sensors with different sampling rates,
we align and the resapmled to 10 Hertz with linear interpolation.
Videos were segmented into clips with six seconds before the is now a good time
questions was asked. In our dataset, driver's responses
are highly balanced with 78% Yes and 23% of No.
We adopted SMOTE to synthezise new samples for the minority class in the
training set to reach an optimal ratio.
Here's the architecture for our multichannel deep convolutional neural network,
PazNet. Channel 1
takes the road video and extract spatial and temporal features by I3D Inception
V1. Channel 2 and 3,
take the pre-processed data from physiological sensors and the CAN system.
We adopted two CNNs as feature extractors,
which are able to capture both temporal characteristics and the internal
correlations among different sensors.
Channel 4 uses landmarks of the driver's face, hand, and body,
which are extracted by Openpose or videos from face and side body camera.
A CNN it's used as well for feature extraction.
We performed early fusion by concatenation of each features and using a
multilayer perceptron for classification.
We combine and compare different feature extraction techniques on both shallow
and deep nonlinear models.
This table shows that PazNet attains the highest performance on both
K-validation and Leave one subject out cross-validation
by combining the data sources and extracting features separately.
Here's an example of our model correctly predict a good moment to interact with
the driver ".
"Is now a good time." Yes.
And here is a moment when our model correctly predict a bad moment to interact
with the driver.
"Is
now
a
good
time".
Because idstracting a driver at the wrong time could be quite bad,
we reweigh the classes of Yes and No training samples
so that we create a more conservative model that can reduce false positives to
zero,
meaning the agent does not predict a good moment to interact with the driver
when the driver says no. But this
graph shows that even most conservative models do still give an average
24 opportunities to interact over a 50 minutes drive,
which we believe that will be still sufficient for real-world systems.
We then retrained PazNet to predict if the moment was safe based on the
third-party annotations using the safety model and the preference model
together. We then build a composite system that predicts first,
if it's safe and then moves on to predicting drivers' preferences,
to ultimately make a final decision on when to interact.
Here's an example of when our model predicts unsafe and the driver says no.
"Is now a good time?".
Still, there are moments where the driver may believe that they can interact,
but our model predicts it is unsafe. "Is Now a good time?"
Yes, I'm just making it turn." In these cases,
we would not have the agent interact.
While this reduces interaction opportunities,
it does not eliminate them because there are still many moments that are both
safe and preferable. In summary,
our research proposes a multi-sensor fusion network to predict when to speak to
drivers based on their preferences and notions of safety.
To support this effort,
we augmented our data with third-party annotations of safety,
and we make this available to the research community. Finally,
we show that fusing automotive,
physiological and visual streams and conservatively tuning our model allows us
to develop an agent that achieves better predictive performance in choosing
opportune moments to interact with drivers.
You can find information and link to the "Is now a good time?" dataset on our
project page. Please feel free to reach out to us,
if you are interested in using the data,
learning more about the modeling approaches that we use or thinking about how
proactive speech agents can tell when to interact with the people based on
contextual factors.