Hello i'm Artem Dementyev and my co-authors are Alex Olwal and Ali Rahimi.
Today i present hidden interfaces for ambient computing enabling interaction and everyday
materials to high brightness visuals on low cost matrix display.
This work was done in Google Research.
As consumer electronics and smart appliances are becoming more common homes are beginning
to embrace various types of smart devices, internet connected technologies that offer
functionality such as music control voice assistant and home automation.
Graceful integration of those devices require adaptation to existing aesthetics and user
style, for example avoiding becoming black screens when turned off.
So there is an increasing desire to create smart devices and appliances which can preserve
the aesthetics of everyday material while providing on-demand access to interaction
and digital displays.
Our vision is to enable low complexity interaction technology that can co-exist with everyday
materials and aesthetics.
Our hidden interfaces are designed to be embedded underneath transmissive materials such as
textile wood veneer acrylic or a one-way mirror and appear on demand for touch-based interaction
by leveraging high brightness low-cost displays.
We show that hidden interfaces can be used to create dynamic and expressive interactions
such as the classical touch-based user interfaces for example a toggle switch or thermostat
control embedded underneath the textile mesh.
We can create a scalable typography such as clock under a veneer material or a caller
id under one-way mirror material and create animation effects such as zooming countdown
or music equalizer.
There are multiple research projects on ambient displays in the ubiquitous computer community.
F or example embedding discrete LEDs under wood material.
Also embedding displays into textiles with optical fiber or discrete LEDs.
Putting Eink displays on objects or clothing.
Screen printing electroluminescent displays on different objects.
Projecting interfaces near the objects or some commercial products have LEDs integrated
under textile mesh such as Nest Audio.
However there are no low cost pixelated displays that can be used underneath materials.
Ambient ubiquitous display need to be low cost low complexity and high brightness.
Eink and LCD don't have enough brightness to penetrate materials.
On the other hand state-of-the-art AMOLED or active OLED displays can be bright however
they are complex and expensive for ubiquitous deployment limiting their use to high-end
devices.
Passive OLED or PMOLED displays are based on a simple design without your pixel memory
which greatly reduces low cost and and complexity.
However they require active display driver circuitry that runs on an update loop that
sequentially activates each row.
With scanline one row can be active at any given time which limits display brightness
and introduces flicker.
To enable high brightness we use parallel rendering instead of tradition and scanline
technique.
The trick is in to in parallel activate as many rows as possible in each operation.
In rectilinear rendering the shapes are broken down into horizontal and vertical lines for
example a square can be shown with just two operations while traditional scanline needs
as many operations as there are rows.
With less operations the brightness of the display can be much higher.
With some limitations such as letters z and x we can render most numbers and letters in
a few operations.
That also applies to simple UI elements such as buttons, switches, sliders and cursors.
In our prototype we use a passive OLED display with 128 by 96 pixel resolution.
For a prototyping purposes we have all the column and row driver routed to a connector
for direct access.
We use a custom pcb with 14 16 channel digital to analog converters to directly interface
those 224 lines to raspberry pi.
The touch interaction is enabled by ring shaped pcb surrounding the display with 12 electrodes
are arranged in arcs.
We compare the brightness of our parallel rendering to a scanline technique on our passive
OLED display and two state-of-the-art small and large active OLED displays.
We tested brightness through six common materials such as wood and plastic.
So the figure you can see shows large rectangle to the different materials on top with our
technique and on the bottom with the active OLED display.
So we found that for simple rectangular shapes you observe 5 to 40 times brightness increase.
The exception was the thick bass wood which didn't let much light through for any rendering
technology as you can see in the figure above.
To conceptualize the findings from our technical characterization with more realistic content
we also evaluate a selection of UIs that can be rendered with three to five operations.
So we did number "2" for single seven segment digit.
Grid of check boxes.
Three progress bars and text string "good life" that shows our ability to render seven segment
strings of arbitrary lengths using at most five operations.
And for this more complex content we observed 3.6 to 9.3 brightness improvement.
We also conducted a user study to understand how legible the text was under different light
condition and materials.
So we looked at the minimum character size that the person can recognize and preferred
the UI element size.
We found that materials significantly affect the minimum legible size.
Smaller UI was possible for mirror material where larger was preferred for wood and textile.
We also didn't observe effect on brightness brightness level on minimum character size
or preferred ui element which might suggest that the parallel rendering technique performs
similarly well in dark and bright environments as you can see there.
We observed three percent misclassification rate and this was due to irregularities of
the material and the the user not being used to rectilinear font.
We also conducted a large scale online survey with 1572 participants to understand the broader
preferences of our implemented interface.
The participants were shown different potential use cases and materials.
So the participant rated the smart mirror use case significantly higher than others.
The user found mirror followed by wood materials to be the most promising and information displays
seem to be more favorable favorite compared to smart controls. In the future we want to
do rendering of content like images and video. We want to create chip on glass display controller
with parallel rendering. We want to conduct longitudinal study in natural environment.
and integrate on screen capacitive touch sensing. In this work we present an interactive hardware
and UI that can blend into range of everyday materials. Our technical evaluation demonstrates
how we achieve unique brightness which allows us to project through wood veneer, textile,
mirror and plastic and our use cases were supported by large-scale surveys and in person lab study.
Thank you for listening.