Hi everyone.
Today I would like to present "InfinitePaint: Painting in Virtual Reality with Passive Haptics Using Wet Brushes and a Physical Proxy Canvas."
The authors of this work are Andreas Fender, Thomas Roberts, Tiffany Luong, and Christian Holz.
InfinitePaint is a Virtual Reality system for painting digitally while retaining the tactile experience of traditional painting.
We use specialized paper that turns black upon contact with water and turns blank again when drying.
This means that users can simply use any wet brush to paint.
Instead of simulating virtual brushes and strokes, we directly capture the physical strokes using an Azure Kinect camera.
The user wears a Virtual Reality headset and can choose the current color and opacity to be used when digitizing the strokes.
All strokes are made persistent and the user sees the digitized strokes with their virtual properties.
This way, the user can paint layer-by-layer, until the painting is complete.
Let's take a closer look at how InfinitePaint works.
At the top, we see the virtual canvas. This is also what the user sees in VR.
At the bottom, we see the camera stream.
The user can pick a color by hovering over the color picker.
When painting a shape, the real stroke is simply black, but the user sees it in the selected color.
To paint layers on top of the existing shape, the user can shift the virtual canvas ...
... so that it is over a dry area of the real paper.
In this example, the user adds shades by lowering the opacity.
While the user keeps painting and shifting, the wet regions of the paper dry over time and the user can re-use them until the virtual painting is complete.
Our processing and rendering pipeline is based on the color stream of the Kinect.
Note that we only use the RGB stream of the Kinect and not the depth information.
The field-of-view of the camera covers the paper as well as the water container.
The perspectively unwarped stream of the paper serves as the input for our processing pipeline.
The Background subtraction receives a grayscale version of the stream and subtracts the current background, which contains all previously painted strokes that are not dry yet.
The Brush tracking component is based on chroma keying and receives a colored version of the input stream.
We wrapped green tape around the brush to facilitate tracking.
Since the color picker is below the virtual canvas, the camera has a clear view on the area that the user painted in just before moving to the color picker.
So neither the brush nor the users hands are covering the painted strokes.
Therefore, the Brush tracking triggers a “capture” event whenever the brush head enters the region of the virtual color picker.
The Stroke extraction then identifies all newly painted strokes and adds them to the list of Persistent strokes.
Furthermore, the Current background is updated so as to subtract the regions that are now wet.
The process then repeats.
The pipeline also provides the imagery to be rendered inside the virtual environment.
We directly display the live-feed around the water container so that users can pick up water at any time.
The Brush tracking turns the brush head into a cursor whenever it is over the virtual color picker.
We render the live-feed of the real canvas including the user’s hand and the brush, with the background subtracted.
Lastly, also render all previously painted Persistent strokes.
This concludes our pipeline.
For more details about the individual processing and rendering steps, please read the paper.
To evaluate our prototype, we invited several artists and hobbyists.
The study lasted around two hours per session and was split into two blocks.
In the first block, we wanted to compare InfinitePaint with traditional painting as well as fully digital painting using a drawing tablet.
This corresponds to our three conditions: Traditional, Tablet, and InfinitePaint.
We provided a reference image of an apple, which the participants had to paint with each interface or medium.
After each condition, the participants filled out different questionnaires.
One of them was the User Experience Questionnaire (or UEQ for short)
The UEQ contains 26 questionnaire items, from which one can calculate and assess six higher-level properties (or “scales”).
The possible scales range from -3, which is worst, to 3, which is best.
Traditional painting was rated quite positively in all scales except for Novelty.
The drawing tablet was perceived quite positively across all scales, but did not excel at any scale ...
... indicating that it is a good all-round solution.
InfinitePaint outperformed the drawing tablet in Attractiveness, Novelty, and Stimulation, but not the other three.
If we look at the categories of the UEQ ...
... we can see that the drawing tablet is stronger overall in the Pragmatic Quality and InfinitePaint is stronger in the Hedonic Quality.
This indicates that a drawing tablet can be useful for general-purpose or high-precision tasks ...
... and InfinitePaint can be beneficial for artistic exploration and expression.
We indeed found significant differences in the UEQ as well as other subjective ratings.
More details about the quantitative results of Block 1 can be found in the paper.
Before the second block, we conducted a semi-structured interview to gather qualitative feedback about all conditions.
Then, in the second block, participants were free to paint anything.
We displayed a browser, so that participants could search for reference images.
The participants also had access to different brushes and to more functionalities of our prototype than in the first block.
This also included functions like Undo.
The main purpose of the free-painting block was to gather additional qualitative feedback.
The participants generally appreciated the realistic appearance of the painted strokes and the tactile sensations, but also identified several limitations.
Please refer to the paper to learn more about the feedback that we received from the artists.
Because we capture the painted strokes directly, InfinitePaint inherently supports different types of brushes and techniques.
In fact, the brush does not even need to touch the paper.
Letting water drip onto the paper and smearing or blowing the drips works as well.
The same is true for creating water splashes by shaking the brush.
We also implemented some additional functionalities that combine traditional and digital capabilities.
For instance, instead of digitizing the stroke with its color, we can use it as a painted mask for applying Gaussian Blur.
In this example, everything below the blue stroke gets blurred, whereas the stroke intensity at each pixel determines the local kernel size.
InfinitePaint can also triangulate and export the digitized strokes so that they can be used in other software.
For example, users can open Blender and import a triangulated mesh of their painting to animate it.
InfinitePaint can also project the painted strokes onto the texture of a 3D model.
This means that users can paint onto 3D models while retaining the passive haptics of traditional painting.
We presented the InfinitePaint concept and prototype.
InfinitePaint supports the capabilities of painting digitally, while at the same time retaining the tactile sensations of painting with a wet brush on physical paper.
We conducted an evaluation with artists and hobbyists who provided quantitative and qualitative feedback.
We showcased several additional digital capabilities enabled by digitizing physical brush strokes.
We hope that our work can inspire future Virtual Reality art mediums that deeply incorporate traditional artistic methods.
Thank you for your attention.