- Thanks, hi everyone, my name is Calvin.
I'm a Masters candidate of Applied Science
at the Human Media Lab, Queens University
Kingston, Ontario, Canada.
I'm also the co-creator of the GridDrone System
which I'll be introducing you all to today.
So this is GridDrones which is composed
of fifteen small tangible Drones
which we call BitDrones on their own,
arranged in a three by five configuration,
but they're not just individual Drones
flying together in a formation,
they're actually a rudimentary programmable matter display.
So Grid Drones is an effort towards the creation
of the ultimate display which I am sure
many of you are familiar with.
It's a step beyond augmented and virtual reality
all the way to a real reality.
But the idea that physical computer interfaces
would one day fully embody digital information is not new.
In 1965, Ivan Sutherland
first conceptualized such an interface.
So the ultimate display would, of course,
be a room within which the computer
can control the existence of matter.
A chair head displayed in such a room
would be good enough to sit in.
Handcuffs displayed in such a room would be confining,
and a bullet displayed in such a room would be fatal.
In 1991, Toffoli and Margolus
coined the term Programmable Matter.
Programmable Matter could be assembled
into clumps of arbitrary size and determined not
by technological limits but by economic ones.
By their definition programmable matter
would consist of small, parallel, cellular automata nodes
capable of geometrically shaping themselves
in 3D space to create any kind of material structure.
This concept of Programmable Matter
was then further refined in Claytronics
in which catoms or claytronic atoms form the material
necessary to construct 3D tangible objects.
The functional requirements for claytronic interface
would be that each catom should be self-contained,
that is, it should contain everything it needs
for computation, locomotion and actuation.
No static power should be required
for adhesion after attachment between catoms,
and coordination should be done through local control,
no external computation necessary.
Catoms should also contain no moving parts.
Grid Drones builds on the functional
requirements of these definitions.
So bringing things back to the present,
this of course is the MIT Media Lab's
Informed Dynamic Shape Display,
which allows for the physical rendering of 3D content
and for tangible interaction with digital information.
There is some exceptional current research
in shape changing displays and modular robotic interfaces
that really tow the line between what could also be
considered rudimentary programmable matter interfaces.
So a prime example of a swarm robotic system
is Harvard's kilobots, which are capable
of acting as a swarm robotic display,
seen in this short video featuring
the construction of a two dimensional star.
Zooids from Stanford Shape Lab are similarly
diminutive robots and are a prime example
of a tangible robotic swarm interface.
They're seen here interpolating
a user's manual input to form a circle.
Another example of a self configuring robotic swarm
is M-Blocks from MIT's C sale,
capable of autonomously creating structures.
Theses small cubes utilizing inertia
from a rotary mass for propulsion
and magnets to maintain physical connections between them.
And most recently, here is a short clip
of Intel's Drone 100 project
which quickly became Intel's Drone 500 project.
Here it was deployed at a music festival earlier this year
for live visualizations and as you can see
it's a highly capable volume metric display
composed of hundreds of individual drones.
Which brings us to the rationale
behind the creation of GridDrones.
To start with there are two approaches as to
how we can interact with digital information in the future.
Either we can augment ourselves,
which is what we're currently doing
for the most part, or we can alter reality,
which is what we just saw several examples of.
GridDrones aims to alter reality,
but obviously this is a very difficult thing to do.
From a programmable matter standpoint,
none of the examples portrayed
fulfill all of the criteria necessary.
Swarms such as zooids are tangible but two dimensional
while Intel's Drone 100, 500 and beyond
are not tangible and are very sparse
due to turbulence and thrust streams from nearby drones.
With M-Blocks and Inform, we have 2.5D displays
which essentially means the 3D structures
must be structurally stable
because the individual elements are not self supporting.
In the design of GridDrones we focused
on the following features:
GridDrones is naturally haptic due to being composed
of self propelled, physical elements that can exert force,
albeit not a large amount but enough
to provide a resistive force when they are being moved.
All primary functions revolved
around the tangibility of GridDrones,
so most interactions are manual.
Certain non-manual functions such as parameter specification
was done using a smartphone app.
Of course being small and nimble drones,
they're entirely self levitating and self powered.
They are capable of 2.5D surface transformations
but because they are entirely mobile and self supporting,
they can also perform 3D transformations
such as rotation, translation and scaling operations.
GridDrones also has the ability
to record tangible interactions
as animations and replay them.
In the implementation of GridDrones,
it comprises of five main parts:
so there are the input devices, the drones themselves,
a UI layer for enacting different interactions scenarios,
the flight controller and the 3D positioning system.
So these are the BitDrones
which make up the GridDrones display as a whole.
What you see on the right is the basic BitDrone core,
while on the left it's pictured securely fixed
inside a light weight carbon fiber and nylon cage.
These cages were designed so that the drone
could be tangibly manipulated and also doubles
as a protective cage for the delicate components inside.
Here's a slightly larger image
that shows the main parts of the drone:
so each has four brushed cordless motors
with translucent propellers for propulsion,
four RGB LEDs to illuminate the drone,
a flight controller with integrated micro controllers
and IMUs as well as Wifi connectivity.
And up at the top you can see
there are five infrared reflective markers
with a unique pattern specific to each drone.
These markers are triangulated
by a Vicon motion capture system
pictured here affixed to the ceiling which provides
position and orientation information of the drones.
The left and right hands are recognized
from a marker pattern affixed to a device worn by the user
or by marker patterns affixed directly to the skin.
This provides a system with information
on when the user is touching or pointing.
A simple smartphone app was created
in order to control non tangible aspects of GridDrones,
such as specifying parameters
or summoning tangible menus out of the dones themselves.
The backbone of GridDrones is a 2.4 gigahertz wifi network.
Ultimately this system is mainly a centralized one
with a flight controller computer running GridDronesOS
connected directly to the network.
This is a system diagram of the GridDrone architecture.
So first the Vicon provides drone positions
to the flight controller.
The flight controller constructs control packets
for each degree of freedom using multi-layered
PID control and then these control packets
are then sent directly to the BitDrones
which can also provide status messages
as queried by the flight controller.
The user then manually interacts with the drones,
or sets parameters using the smartphone app.
This information passes through a UI layer,
which runs the various interactions scenarios
and specifies the necessary spacial arrangements
of the drones in space.
So as I highlighted earlier,
the point of GridDrones is it's a tangible display,
which accepts both unimanual and bimanual input.
There are four ways in which
to group drone elements using manual selection.
Unimanually a user can either select all,
perform a contiguous or discontiguous selection
or perform a traditional lasso select.
Bimanually a user can also select
a rectangular set of boxels between touch locations.
So here a user double taps a single BitDrone
to select all elements and group them.
GridDrone behavior's then specified from the smartphone app
in this case parameters were set
in order to create a pyramid shape
when a control point was dragged upwards.
GridDrones also allows
for arbitrary groupings within itself.
Here the user performs a discontiguous selection
where only the corners of the grid
are unimanually selected and grouped.
A lasso select is also possible using a ray cast
from the vector as indicated by the index finger.
The lasso selection process is terminated
by an additional click with the thumb.
3D transformations are done entirely
through bimanual interaction with the GridDrone display.
With a bimanual interaction, the user can either
rotate the GridDrones about all axis
or translate along all axis in 3D.
Beginning of bimanual interaction is as simple
as the user reaching out and grabbing two drones.
Moving them in space relative to each other,
allows for 3D rotations about all axis
and additionally lateral translation is possible
by simply moving the two drones together
in the chosen direction of movement.
So there are several examples that we're constructed
and tested with the GridDrones
including 2.5D NURBS surface modeling,
tangible animation design and flying LEGO bricks.
The most obvious application
for GridDrones is in 3D modeling.
The NURBS surface application runs
in the GridDrone UI layer which allows the user
to program a topological relationship
between groups of the drones.
These relationships are programmed
using the smartphone app where a relationship
can have a strength ranging between zero to 100%.
100 would represent a hard material relationship
where as 25% would represent
more of a soft material relationship.
Users can program multiple overlapping groups
allowing for complex behaviors to emerge.
So here's the NURBS surface application in use.
First the use double taps to select all elements,
then a topical topological relationship is applied,
roughly 50% in this case.
By pushing and pulling the BitDrones
which represent control points,
deformations can be applied dynamically
throughout the entire surface.
Aforementioned 3D transformations can subsequently
be applied in the form of 3D rotations and translations.
Here a ray cast select is used
and topological GridDrone relationship's set.
Control points do not necessarily have to be
individual drones but can consists of a line of drones
as selected though a ray cast from the index finger.
In this case, a ray cast selection is used
as a line control to construct an archway.
It's also possible to animate
these GridDrones models and replay the animations.
Using the Grid a user can program an animation
by setting a topological relationship,
beginning a recording and then moving the BitDrones.
A recording menu is summoned from the smartphone
to either record a new animation with the red button,
or playback a previously recorded one with the green button.
The recording can be played back through the system,
and the user can then move it around space
using aforementioned 3D translation.
Because manual interaction would conflict
with the live animation, the model has to be grabbed
by a closed fist gesture and then translation enacted
through the extension of the index finger
and then motion of the hand.
So in February, we worked with LEGO
to create a tool for creating and animating
robotic structures based on BitDrones, GridDrones.
Although the drones are pretty safe , we had to make sure
such an experience was absolutely child-proof.
Both for the sake of the kids as well as the drones.
We determined that separation from GridDrones
behind a window was the best option.
To substitute tangible interaction
with the drones themselves we created
an embodied controller to directly
control them from behind the glass.
Each embodied controller is connected to the GridDrones'
wifi network and has two integrated IMUs, one for each wing.
They're also equipped with lights, and an internal battery.
The GridDrone's UI layer uses the IMU data
from each wing to determine the absolute orientation
of each wing as well as the relative orientation
of the wings to each other.
The body of the controller is constructed
of a modified LEGO baseplate,
on which LEGO bricks can then be attached.
A scanning table was also constructed with a USB camera
and an Intel Nuc PC running a custom open CV application,
to determine relative LEGO brick location
as well as their color.
Using an embodied controller,
the user can scan a LEGO creation
and then the system will do its best to recreate it.
Once assembled, the system allows for full embodied control
with direct mapping from the controller
to the GridDrones themselves,
the bending of the controller wings
will result in the relative bending of each wing
in the flock while rotating and tilting the controller
will similarly result in a tilting and bending of the flock.
We took the system to the 2018 LEGO World expo
in Copenhagen where approximately 250 children used it.
Each child responded positively
to the exhibit without exception.
Just last Thursday, we took the system to Madrid
where it appeared on a live show called El Hormiguero
and was piloted by a guest star, Rick Asley.
This demonstrated that the system is highly reliable
but it does have some issues.
Which brings us to the limitations of GridDrones.
This was not an unfamiliar scene.
(audience laughing)
So namely the battery life is relatively good
for small drones at about seven minutes
but its not long enough for sustained use as a display.
This is simply limited by modern technology,
modern battery technology and there's not a lot
we can do about it right now.
The resolution of GridDrones
is limited by the cost of the drones,
the physical volume of the interaction space,
as well as the size of the tract volume
as enabled by the Vicon motion capture system
which also contributes to the cost.
Due to the proximity of the drones
that can't fly over top of one another
as they cannot stay aloft in the thrust streams
of drones over top, this creates issues
with true dense 3D models.
Smaller drones would be better
so that the density of the display could be higher,
but the drones are completely custom designed by us,
custom built, and also have to remain serviceable.
So one way to circumvent the issues with battery life
would be to dynamically replace drones
as their batteries run out, if we can recharge drones
at the same rate that they require replacement
then this would be a solution
for longer use cases of the display.
Wireless power transfer would negate the need
for better batteries or dynamic replacement
but is pretty far off in terms of void
and power requirements for this project.
Smaller drones will increase the display density
of GridDrones which we're currently working on
and being able to hover over each other
will allow for true 3D tangible construction.
3D group selection is only possible in a 2D grid at present
and alternate selection techniques will have to be explored
for true 3D drone configurations.
So we're also working on a physical mid-air connection
which will allow for a much denser display.
Pictured here is just four drones but we've done it
with as much as six and we're hoping
to scale that number up but there's a lot
of control system issues that come along
with flying larger flocks of connected drones.
And that's it, Thank you.
(audience applauding)
- [Speaker] Alright so lets line up for some questions.
Dan is already over there, I knew it.
- [Dan] Hi Daniel Ashbrook, University of Copenhagen.
This is super cool, I really like the idea of being able
to actually grab and physically manipulate these things.
But something that I was thinking about is
in the hand wavy future, when you have a thousand
of these things they start to get in the way of your body.
And I'm curious on how do you envision being able to
manipulate a drone that is in the middle of the flock
that you can't actually physically reach.
- Absolutely, so that's kind of why we started
exploring things like ray casting for selection.
Obviously there's a lot of work done with 3D selection
in virtual reality, we find that there are issues
with bringing this stuff from virtual reality studies
to a physical thing as you mentioned
but that definitely warrants much further research
and studies with this system.
With something like that we might envision
the user actually going into the display itself,
doing interactions that are much smaller scale
and then taking a step back and maybe interacting
guesturaly with something of a much larger magnitude.
- [Dan] Well Thank you.
- [Female ] Hi I'm new from MediaWork.
I really like your project, but like when I saw
the 3D display through the air I saw that you're trying
to display some images using LED color changing,
but have your considered instead of using structure,
I mean because you using strong bit to maybe you
use a structure, a skeleton structure but what would be
another possible way to display an image
on one mode of the drone or display or shape
instead of one LED colors.
- Well to be honest we haven't thought too much about that
just because what we're really focused on is the idea
of each drone representing a catom,
representing a claytronic atom so we're more concerned
with minimizing the overall size of the drones
and increasing display density than actually trying
to provide more contextually relevant information
on one drone, so the focus again is really
on managing all of the drones together
instead of creating higher display resolution
on the drone itself which is kind of why
we just stuck to one color so that
we would remain in that vein of research.
- [Female] Aright thank you.
- [Speaker] Alright lets thank the speaker once more and
(audience applauding) we're wrapping up the session.
Just want to say