Hi everyone, and thanks for joining us.
My name is Clement and with me virtually are my co-authors, Peter and Ellen from the ATLAS
Institute at the University of Colorado Boulder.
We are excited to share our work - Tangible Interfaces with Printed Paper Markers -
with you today.
In this research, we explore the unlikely intersection between computer vision, and
the material affordances of paper.
Specifically, we explored physically manipulating paper with printed AR markers,
And investigated how this can support the rapid making and sensing of
inexpensive tangible interfaces.
This might sound rather abstract, so let me illustrate this with a simple demonstration.
To begin, we print and cut out a strip of paper with an AR marker that is divided into
two halves.
This strip is then folded into a three dimensional structure
that brings the two marker halves together.
We adjust a stage, and position a camera below it.
This camera observes the underside of the paper push button.
When the button is pressed, the marker is completed and detected
by the computer vision system.
When released, detection is lost.
We can therefore use marker detection to sense button interactions.
Our research on tangible interfaces with printed paper markers was driven by a few inspirations.
We are attracted to paper as a material for tangible interactions.
Paper has been widely explored by the HCI research community for different contexts.
For instance, paper is an accessible material especially for children,
and researchers have investigated paper as a platform for STEM learning.
Paper also presents many structural affordances for three dimensional interfaces,
Such as paper as a medium for shape-changing interfaces.
We have also previously explored paper for three dimensional interfaces in our lab.
In that project, we investigated sensing physical deformations through kirigami techniques and
carbon-coated paper.
Tangible interfaces made from paper are light and inexpensive.
However, for them to work we usually need to tether them to complicated circuits and
bulky electronics.
And so, with that challenge in mind, we looked into computer vision
for sensing tangible interactions.
In particular, we focused on computer vision with fiducial markers, such as the setup employed
by reacTable.
In such systems, many physical tokens can be tracked with a single camera, and these
tokens are also untethered and free-to-move.
With those inspirations in mind, we started our exploration into tangible interactions
and printed paper markers.
Before jumping into what we did, let me describe the computer vision setup that we used.
This setup has three main components.
First, we have an adjustable stage made from clear plastic.
Paper interfaces are placed on this stage.
Second, we have a camera that observes the bottom side of this stage.
Finally, we have a computer which the camera is connected to.
This computer runs the computer vision script.
For our exploration, we employed the ArUco marker library.
There are a variety of different marker libraries out there, and we chose the ArUco library
as its markers are graphically simple and
also offered an efficient and robust detection algorithm.
The library we used offers up to a thousand unique markers.
In our exploration, we focused on experimenting with tangible interactions that are facilitated
by constructing and deconstructing markers.
As demonstrated earlier, we explored making simple push buttons
out of paper and printed markers.
We realized that we could create push buttons with different forms that all employ the same
sensing strategy of completing a marker.
In this case, we have a symmetrical button.
But, by folding the strip of paper differently, we arrive at an asymmetrical button.
We also explored pushing the limits of paper printing.
In this case, we printed a pattern on the folding lines of a paper spring.
When this spring is fully compressed, the printed pattern assembles into a marker which
is then detected by computer vision.
Besides sensing tangible interactions by detecting marker construction,
we can also sense tangible interactions when deconstructed markers lose detection.
In this example, we print a series of markers on a stretchable paper structure.
At rest, all markers are detected.
As this structure is stretched the middle portion expands faster than the sides.
As such, the middle marker loses detection first,
followed by the side markers when stretched further.
We can therefore use the loss of marker detection to approximate how much this paper structure
has been stretched.
So, that was a snapshot of our exploration.
And after playing around with these printed paper marker experiments, we took a step back
and asked ourselves why this might be important, or useful,
to other designers, makers, and researchers.
Paper is an accessible resource and is readily available to most makers.
On top of that, paper presents accessible fabrication workflows.
For instance, it does not require expensive machines to cut, fold, and bend paper into
three dimensional structures.
It is also really easy to print graphics on paper,
and we can use this to explore the visual features of an interface, as well as instrument
the inputs with printed fiducial markers, As we did in this research.
This enables multiple paper inputs to be detected with just a single camera.
Because of paper and the fabrication processes it supports,
it was easy to document our exploration.
In the pictorial, we embedded fabrication files for each prototype as vector drawings.
For anyone who wants to follow along our exploration, these files can be accessed by opening the
publication in a vector drawing tool like Adobe Illustrator.
To showcase the potential of the printed paper markers technique,
We proposed and built a few different applications.
For our first proposal, we used printed paper markers to build interactive data physicalizations.
For the interactive bar chart, each bar is a three dimensional push button.
Pressing a bar triggers the system to read out more information about that specific datapoint,
while pressing on two bars reads out comparative information between the two points.
We also made an interactive line chart with similar features.
Printed Paper Markers can support industrial designers to rapidly test interface layouts
as well as explore product color and visuals.
For our second proposal, we used printed paper markers to build a prototype of a MIDI controller.
This prototype has fourteen push buttons, two sliders, and a knob.
To create a slider, we connected a series of push buttons with unique markers.
Pressing down on different parts of the slider will complete different markers.
We use this to approximate the slider position.
For our third proposal, we made a digital game played with a delicate and disposable
game controller.
In this game, players take control of a boat which they paddle by pulling the controller.
However, we intentionally designed the controller to tear if a player exerts too much force.
Tearing destroys a marker and disables the controller,
and in that case, players will need to replace the broken controller with a fresh one to
continue playing the game.
So, I want to wrap up our presentation by discussing a few limitations of the printed
paper markers technique, and some plans for future work.
One of the biggest limitations we faced during this research was how sensitive computer vision
is to environmental conditions, especially ambient lighting.
And in our exploration, we faced this tension between what we see as humans, and what is
detected by the computer vision algorithm.
Human vision naturally complete patterns and fills in the gaps.
But these same discrepancies on the other hand are amplified by the computer vision system.
The detection software that we developed therefore became an invaluable debugging tool.
And the more we used it, the better we got at figuring out the nuances of computer
vision and apply it to our designs.
We are currently working on developing a networked mobile platform for fiducial marker detection.
We believe that mobile platforms will improve the portability and accessibility of printed
paper markers, enabling it to be more effectively deployed
in different scenarios, such as classrooms or museums.
And so with that I have come to the end of our presentation.
Thanks for watching, and do reach out to us if you have any feedback or questions!