Good afternoon, everyone.
My name is Sangyoon Lee.
This study was done with my great collaborators, Youn-kyung Lim, and Geehyuk Lee.
In this study, we introduce MirrorPad which is a novel interface device contained in a
laptop for direct pen interaction.
Today, Let me show you our work and interesting findings.
In the conventional laptop environment, users interact with computers mainly using keyboards
and touchpads.
However, some activities can be performed more effectively and naturally with a pen.
Many popular applications provide functions that enable users to make notes and sketches
with a pen.
Users can be more productive with these applications if they can use a pen in the laptop environment.
The computer market sees various ideas to support pen interaction in the laptop environment.
The ﬁrst option is to use an external pen tablet. The second option is to use an external
pen tablet with an LCD display.
Both options may not be practical, when the portability of a laptop is considered.
Though users often use a laptop in an ofﬁce, they may also need to use it elsewhere, such
as in a classroom or while traveling.
Carrying a pen tablet or a pen display everywhere just for pen interaction is not convenient.
The third option is to embed pen hardware in the laptop display.
In this case, pen interaction is direct, and users do not need to carry an additional device.
The computer market sees many pen enabled laptop products recently.
An increasingly popular idea is a convertible laptop.
These laptops are convertible between laptop and tablet conﬁgurations and can support
both keyboard and pen interactions well.
The laptop conﬁguration comprises a vertical touch screen, a horizontal keyboard, and a
touchpad.
This conﬁguration enables efﬁcient text entry with the physical keyboard and precise
pointing and skilled manipulation with the touchpad.
In contrast, pen interaction is limited, because users should use a pen on the vertical screen.
The tablet conﬁguration provides users with a wide horizontal touchscreen.
This conﬁguration enables users to perform natural pen interaction like they do on paper.
However, text entry is not efﬁcient, because the virtual keyboard blocks half of the screen
and provides no tactile feedback.
As the two conﬁgurations complement each other, users can switch between the two conﬁgurations
depending on the current activity.
However, the problem is that some activities need both key-board and pen interactions interspersed
in the same workﬂow.
In such cases, users may need to switch between the two conﬁgurations frequently.
Conﬁguration switching costs time and physical efforts.
In addition, it can make users lose their current context, thereby incurring a high
mental load.
In short, the idea of a convertible laptop may not work well for activities that require
interspersed keyboard and pen interactions.
To complement the weakness of the convertible laptop, we proposed the idea of MirrorPad.
Located where the usual touchpad is, MirrorPad is both a touchpad and a pen-enabled display
that mirrors a region of interest on the main display.
Users can use it as a normal touchpad.
In addition, they can use a pen on MirrorPad to write on the mirrored region.
I will explain the interaction of MirrorPad in more detail.
Users can use MirrorPad to control the mouse cursor and perform multi-finger gestures.
At the same time, MirrorPad mirrors the content of a region of interest in the main display.
For moving and resizing the region of interest, MirrorPad has a quasi-mode.
When the pen is in the hover tracking range of MirrorPad, the quasi-mode is activated.
In the quasi-mode, the normal touchpad functions are disabled.
Users can use touch gestures to move the region of interest.
In addition, users can use touch gestures to resize the region of interest.
To prototype the MirrorPad concept, we used two Microsoft Surface Pro 3 tablets and an
Apple wireless keyboard.
The bottom tablet and the Apple keyboard emulate the bottom part of a laptop, while the upright
tablet emulates the main display.
The bottom tablet transmits all events from MirrorPad to the upright tablet, and the upright
tablet transmits the captured image of the region of interest to the bottom tablet.
We conducted the user study with the ﬁnal MirrorPad design to verify how well MirrorPad
would support a workﬂow with interspersed keyboard and pen interactions.
Therefore, we decided to compare MirrorPad with the two conﬁgurations of the convertible laptop.
To simulate a workﬂow with interspersed keyboard and pen interactions, we considered
a programming lesson situation, in which a user uses keyboard interaction for typing
code to solve problems in an IDE and pen interaction for handwriting answers to questions provided
in a PDF ﬁle.
Then, we designed Keyboard and Pen sessions.
In the Keyboard session, a user uses keyboard interaction to ﬁll in blanks in a program
to make its output equal to a correct answer.
In the Pen session, a user predicts the output of a given program and writes it on the page
with pen interaction.
We recruited 12 participants, who have used a laptop supporting a digital pen for at least
one year.
The participants were asked to perform 10 sessions under each condition.
The figure shows mean completion times of the two sessions.
In the keyboard session, Tablet condition was significantly slower than both MirrorPad
and Laptop conditions.
In the pen session, Laptop condition was significantly slower than Both MirrorPad and Tablet conditions.
Based on the completion time results, we concluded that MirrorPad was as efﬁcient as Laptop
condition during keyboard interaction and showed a performance similar to that of Tablet
condition during pen interaction.
In the user study, we observed differences in the annotated results among the three conditions.
In the Laptop condition, all participants could not write horizontally due to gravity.
Most of their annotations tended to start at the top and end at the bottom.
In the Tablet condition, the users interacted naturally with a pen and made many notes.
In the MirrorPad condition, the users performed pen interaction similar to the Tablet condition.
However, we observed that users made notes in a relatively narrower region than that
in the Tablet condition, possibly because the size of MirrorPad was smaller than that
of the main display.
Based on the results and observations, we concluded that the ﬁnal design of MirrorPad
turned out to support the workﬂow with interspersed pen and keyboard interactions effectively.
The contribution of this study is not only to present an interface that turns a touchpad
into a pen-enabled touchscreen, but also to experimentally validate MirrorPad technique
that effectively supports interspersed keyboard and pen interactions on the laptop.
This is the end of my talk. Thanks for your attention.
There are more interesting contents in the paper, so if you are interested in my presentation,
please read the paper.