Hi everyone, so today I'll be talking about
our ToCHI paper creating and augmenting
keyboards for extended reality with the
Keyboard Augmentation toolkit or KAT.
My name is Mark McGill.
I work at the University of Glasgow and
this was a result of a collaboration
between University of Glasgow and
Logitech Design Lab in Ireland.
So extended reality headsets are
increasingly being used to drive
productivity oriented experiences
from apps such as big screen,
pictured here in VR. To Meta's
infinite offers functionality
in Quest headsets built around
progressive web apps and Android apps.
Crucial to the use of such
workspaces is text entry.
This predominantly requires a
high bandwidth input modality,
namely the physical keyboard.
Logitech and Meta have worked on
keyboards whose position and orientation
can be tracked by XR headsets,
enabling full seated desk
based productivity in VR,
and this functionality can be used
to render an interactive digital
twin of physical keyboard in VR,
or it can be used to selectively
show areas of the pass-through feed
pertaining to the keyboard,
either way enabling direct use
of this high bandwidth input
peripheral in a VR or XR headset.
But one advantage that has been somewhat
overlooked is the capacity for these devices,
not just to present virtual
displays or 3D content to the user,
but also to augment or alter the peripherals
and surfaces that they're interacting with.
For example, if we know where the keys are,
we can visually augment the
keyboard in any way that we see fit.
For example, rendering active shortcuts or
visualizing different overlays or mappings.
And if we have hand tracking our
capacity to interact with the keyboard,
also greatly increases.
Able to support interactions,
for example on or around
the keys of the keyboard.
Touch display keyboards was
perhaps one of the earliest
examples of augmented keyboards.
This was enabled using projection
on a physical keyboard,
and the project supported a
plethora of interactions,
including multi key buttons and
temporary keyboard based menus,
all visualized directly on
physical keyboard itself.
For interactions on and around
the keyboard project such as GestAKey,
FingerArc and FingerChord
have all exploited some capacity to track finger
movements or receive additional input
that could enhance our capacity for
expression based on an interaction
with a single key on the keyboard
going beyond binary keypresses alone.
However, the majority of the previously
mentioned projects shared common drawbacks,
particularly around the hardware
needed to enable these interactions.
So capacitive sensing and per-keycap
displays are common themes here,
but integrating such technologies and
keyboards is expensive, difficult,
and not particularly sustainable
given we all have perfectly good
keyboards right now.
XR headsets however,
offer offer us the possibility of
driving such activities without
the need for additional hardware
beyond the physical keyboard itself.
So Reconviguration here demonstrated this
when they rendered different character sets,
and different UI is on a
tracked physical keyboard in VR.
And building on this,
the Keyboard Augmentation Toolkit
is our first attempt to support
the creation and exploration of XR
driven keyboard interactions where
the very form and function of the
keyboard can be altered. So our aim
was to have an open source toolkit
that would enable practitioners
to build novel augmented keyboard
interactions and replicate and transpose
previous research where applicable,
taking advantage of the combination of
the tracked keyboard and XR enabled,
sensing such as hand tracking.
Our tool kit works with the Unity gaming
engine and supports the ability to
define augmented keyboard interactions
through creating new keyboard mappings.
So a keyboard mapping effectively
describes what each key should
look like and what it should do,
and this can be as simple as having
a mapping to show a different
character set versus more complex multi
level hierarchies of key functions.
For example, a Photoshop mapping might
visualize its keyboard shortcuts by default,
but overload keys with additional
functionality when the color picker is open.
And each of these mappings is made up
of one or more key elements, where a key
element notes the set of interactions
to be bound to a set of one or more keys.
For hierarchical mappings,
here we can see that the keyboard
mapping is presenting emoji input where
the keys along the top of the keyboard
all activate child keyboard mappings
for different subsets of emojis.
This gives us the flexibility not
just to define new interactions,
but also have those interactions
coexist with others non destructively.
Key elements can display content
in 2D or 3D across one or more keys
and can receive and react to an
extensible variety of interaction
event triggers from simple binary
keypresses to hovering above the key,
or GestAKey style
swiping or taps of the surface of the key,
and so on, all without needing any
integrated capacitive or pressure
sensing in the physical keyboard.
So this is all driven by the
hand tracking in relation to
the tracking of the keyboard.
These key elements can span multiple keys,
creating interactive regions that
support both discrete and continuous
inputs on and around the keyboard,
again driven by XR-enabled hand tracking.
So here you can see a user
interacting with each multi key
region with a mid-air keyboard.
And these keyboard mappings can
be applied to any keyboard layout.
Be it a real physical keyboard that's
being tracked by an XR headset or
a mid air virtual only keyboard.
And our toolkit provides assistance
in defining and generating these
keyboard layouts, for example,
based on an existing digital twin 3D model.
I'm not going to go into
great detail about KAT here,
except to say that this is an
open source extensible toolkit
that provides a strong foundation
for creating a wide variety of
augmented keyboard interactions.
To give an example,
this is a screenshot of the new
AZERTY keyboard layout that was
imported into KAT and visualized
on an existing physical keyboard,
a version of the Logitech mixed reality
keyboard for Windows Mixed reality. And
it becomes straightforward to import,
visualize and switch between different
keyboard configurations on the fly here.
Meaning the physical keyboard can adapt
to the needs of the user with the
interface even extending to the keyboard
rather than simply receiving events
from the keyboard.
And here we can see the same new AZERTY
layout applied to two very different
configurations of mixed reality keyboards
based on prior research. On the left
here this was a paper looking at mid
air keyboard input in VR; on the right,
this paper was looking at keyboard
input where the virtual interface
was aligned to the surface of
a table and we can see that the
mapping is applied equally to both
of these different layouts.
KAT facilitates the creation
and further exploration of
augmented keyboard capabilities.
Our hope is that this toolkit
represents the first step towards
having augmented peripherals
supported at operating system level
on XR devices such that we might
appropriate and augment keyboards,
mice and touchpads as part of our
daily productivity experience
when using our headsets.
Our paper goes in far more detail
about the toolkit and our initial
explorations of this space.
So please see the ToCHI
paper for more details.
Thanks.