- From ShanghaiTech University, today we introduce
SweepCanvas, for quickly 3D prototyping on an RGB-D image.
This is a co-work with Zhejiang University,
Shenzhen University, and City University of Hong Kong.
As you know, 3D modeling has a high demand
in 3D printing and augmented reality.
Conventional 3D modeling tools,
like Maya and Blender, they support
powerful and general purpose 3D modeling.
However, they all require
tedious labor and professional skills.
Our aim is to offer users a simple,
yet effective, way to quickly create 3D models.
The models should also fit into the real world.
This idea is called modeling context.
So we present a sketch-based interactive tool
to quickly produce conceptual 3D models
atop an RGB-D image.
On the left, in the point cloud computed
from the RGB-D image, and on the right
is our model in the point cloud.
Here are some previous works of sketch-based,
free-form 3D modeling, by interpreting user sketches.
And here are some works of 3D sketching
that directly map 2D sketches into 3D
by lifting the 2D sketches.
However, these works do not use any context information.
The idea of modeling context is not new.
Second Skin requires a complete 3D model as input.
Insitu require computed point cloud.
SmartCanvas requires a set of pre-defined 3D plans.
While our input is a single, unstructured RGB-D image,
which is lightweighted, realistic, and easier to acquire.
Depth sensors like Kinect are cheap and easy to carry,
and depth sensors has also been adapted to cell phones.
The new iPhone X is believed to have
the front depth sensors that is essentially
a mini version of Kinect One.
It is getting easier to take RGB-D photos.
But there are still challenges to use a single RGB-D image.
One challenge is to understand the scene.
Another challenge is to understand the sketch.
For understanding the scene, we extract
planes and 3D features, like corners and edges.
For sketch, we made an assumption
that each stroke lie on some spatial planes.
We then proposed multiple candidate planes
for user to select, to make sure we get
the correct results.
And before we go into algorithm details,
I would like to show you a short video demo
of our work first.
So this is interface.
You can create an canvas using two strokes,
profile stroke and the trajectory stroke.
You can also select an existing stroke
and create another canvas.
You can also select two existing strokes
and combine them to create a canvas.
We use the right button for rotation
and the view for zooming and zoom out.
And we also support auxiliary operations
like cutting and editing, to decorate our canvas.
And we also use circle for profile strokes.
You can check the symmetry button
to create symmetry shapes like the vase.
And you can also create canvas from an existing canvas.
And on the right side is a candidate panel.
And through the candidate panel you can
choose the best canvas that you want.
Because there are many possibilities
for 2D to map to 3D, so we propose candidates.
And then you can also paint the canvas
to decorate the scene.
And because our image is taken in the real scene,
we can't compute the actual 3D position
from the RGB-D data with camera parameters.
Right now, the scale of the models
are controlled by the users.
These are some basic 3D elements
that constitute our system.
These are called s-canvas.
The red and blue curves are user strokes.
We call them profile stroke and trajectory stroke.
The surfaces are generated by moving
the profile's curve along the trajectory curve.
In this image, (f) is derived
using the same strokes in (e),
with symmetry enforced.
Here is the animation for creating basic s-canvases.
Given an RGB-D image, we first analyze,
in which we extract planes and 3D features
like edge and corners from the RGB-D image.
The second stage is a modeling,
in which we create the s-canvas.
Given the user strokes from the planes
extracted from the context, we select
a base plane for the profile stroke
and a reference plane for the trajectory stroke.
We project the strokes on the plane
to get 3D positions in the context.
Then we sweep the profile stroke
along another to get an s-canvas.
We also consider several snapping cases
to make our drawing experience more fluent.
Our system also allows several auxiliary operations
to enrich and smooth the modeling process,
like editing and painting.
As mentioned before, we use planes
to understand the scene.
We also need planes to understand the strokes.
Because we made an assumption
that strokes like on some spatial planes.
Like in this image, the red and blue stroke
lie on the red and blue plane.
To extract these planes from the RGB-D image,
we use mean shift to cluster the point cloud
based on their normal color and depths.
Points in one cluster may have similar color and normal.
Then we run Ransac iteratively in one cluster
to get the 3D planes.
Besides planes, we also need 3D features
to help us understand the scene
and find in-context relations.
Like in this image, the yellow stroke
is snapped to the edge of the table.
To find these 3D features,
we detect the edge and corner in 2D,
and then consolidate them in 3D.
For 2D edge, we consider it as a 3D edge
if it agrees with one of the boundaries
of extracted planes.
Similarly, for a 2D corner point,
we consider it as a 3D corner point
if it is close to a plane corner.
So this process will retain a clear set of 3D features,
like in the right image.
Now I'm going to talk more about the modeling stage.
Remember the assumption that the profile stroke
and the trajectory stroke, they both lie
on some unknown spatial planes.
So different choice of planes can lead
to different results of canvas.
Two strokes on the left image can create
a canvas that either lies on the ground
or is perpendicular to the ground.
For the extracted set of planes
and two user input strokes, we formulate it
into an MRF labeling problem.
Here, E is the energy function,
and p is extracted planes, and s stands for stroke,
s-p is a profile stroke, and s-t is a trajectory stroke.
In this function, we compute the energy
for assigning plane i to profile stroke,
and plane j to trajectory stroke.
In the unary term, we measure how likely
a drawing stroke lies inside a plane.
And in the binary term, we encourage
the selection of planes which are approximately as ordinal.
And for more details of this function,
please refer to our paper.
According to the scores from the energy function,
we choose our first set of planes as candidates,
and push them into the candidate panel.
The selection process will give us
a pair of planes which exist in the context,
but the actual position of these two planes
can deviate by some offsets along the normal directions.
We need to find an attachment point that the base
and reference plans pass through.
We exam intersection point of the two strokes,
see if they're close to any 3D corners.
And if not, we search for edges,
to see if any stroke aligns
or partially aligns with any 3D edge.
If none of the above cases is found,
we set the attachment at the most front point
in the local window of the current viewing direction.
We'll rank these plane pairs according to their scores
computed according to the energy function.
Then we push them into the candidate panel
for the user to select, in case
the desired result is not in the main window.
To facilitate more fluent drawing experiments,
our system supports three snapping cases.
First is snap to a 3D plane.
The second is snap to an existing
3D corner or 3D edge.
We can also snap an existing s-canvas point.
Here we show more operations,
cutting, painting, and editing.
By editing, you can select the profile
or trajectory stroke to edit the canvas.
Here is a complete demo of our system.
For the user study we conducted with 10 participants,
among them, three were experienced users in 3D modeling.
Four are beginners, and three people had no experience
in drawing or modeling.
We first teach the users to use our system
and give them 10 minutes to practice
and try out our system freely.
We designed three tasks for the user study.
In the first two tasks, we asked them
to reproduce a given examples, as showing here.
In the third task, we can draw freely on our given scenes.
This is a statistic of our first two tasks.
It can be observed that as the example becomes more complex,
the time required for each example
increased roughly linearly.
The candidate selection operation kept at a small ratio,
it shows a robustness of our system.
Our system also allows a user to manually specify
and manipulate a 3D plane to serve
as supporting plane for the strokes.
This operation is called plane manipulation.
According to the statistic, this operation
is also rarely used.
In the third task, each participant
was invited to draw three examples
on top of a given scene.
Here are some models created by our participants.
In addition we did a basic implementation
of the baseline application.
User need to manually select each plane
to draw the profile and trajectory curves,
without stroke inference.
We disable any automatic inferring and snapping
features in this mode.
Then, users has to manually translate
and rotate the plane to a desired position.
We think this mode mimics a Sketchup interface
which use manual or pre-defined planes
to anchor swept surfaces.
It can be concluded from the statistic
that such a baseline mode would lead
to a longer time and more view change
plane selection operations.
And because we did evaluation on the simple example,
the time used to create the scene is below four minutes,
even for manual plane-specification mode.
But it is still two times longer than our method.
There are some limitations for our work.
Because we extract planes from 3D feature,
because we extract planes and 3D feature
from the RGB-D image, we cannot create
models when a desired 3D supporting plane
does not exist in the context,
or it cannot be added by our algorithm,
like a floating cube.
In such cases, a user can perform
the auxiliary operation of plane manipulation,
create such a 3D plane.
Also remember the assumption at the beginning
that the stroke lie on some spatial planes.
This assumption allows us to understand the user sketch.
Thus we cannot handle cases that the trajectory curve
doesn't lie on some spatial plane.
Like the spring, which has a spiral trajectory curve.
To conclude, we introduce an effective
sketch-based in-context modeling system.
We also present a plane extraction algorithm
for RGB-D images, and an optimization algorithm
for automatic inference of 3D strokes
from pair of 2D strokes.
In the future, we plan to experiment
with more different media
such as single images or videos.
We also believe such modeling tools
can be adapted to other applications like VR and AR.
And we thank the reviewers and many
user study participants and the fundings.
Thank you for your attention.
(applause)
- [Audience Member] So, nice work.
So it seems you only add something
to existing thing, and you do not allow user
to modify existing depth image, right?
- I'm sorry?
- [Audience Member] (laughs) Can you edit
the depth image?
- We can edit the canvas, if you mean that?
- [Audience Member] What I mean,
remove existing chair, or rescale existing chair?
- If there is something existing in the scene,
we cannot remove it because we compute it
from the point cloud, so it's in the context.
We can edit something in the scene.
- [Audience Member] So yeah just suggestion,
might be interesting if you can also edit
existing things. - Yeah, yeah.
- [Audience Member] I attend (mumbles) University.
So, your study was done with a mouse, is that correct?
The input device that the participants used
was a mouse, so was it a--
- Yeah.
- [Audience Member] Yeah.
Do you think it would make, would it be more fluid
using a pen interface or some other type of device?
- Yeah, we have tried to use a stylus pen,
and in our study there are people who used,
they can choose to use a mouse or stylus pen
based on their familiar,
but we used the right button for rotating the thing,
so it's better to use a mouse.
- [Audience Member] Thank you.
- Thank you.
- [Audience Member] I have a question.
(talking in background)
- Yes, that's the good question.
We actually considered extracting curved surfaces,
and because right now, only the planes,
the planes cannot create very complex models.
Like in that limitation, if we choose curved surfaces,
we can create more complex models.
We're doing that in our future work.
- [Speaker] We have no more questions,
I thank our speaker today.
(applause)
- Thank you.