Good day everyone.
My name is Thanh.
I'm a PhD student in the UTS: Robotics Institute.
Today I m going to present my paper title Investigating the Effect of Sensor Data Visualization
Variances in Virtual Reality .
First, I want to introduce and talk about the motivation this paper.
I'm a part of the research team working on HALO, a research collaboration project between
Ausdrill, IMCRC and UTS.
The system we are creating aims to assist with rock scaling operations that workers
currently do manually. You can see the image of our system on the right
Because robots are currently not very good at eye-hand coordination and making high-level
decisions yet, the system cannot be fully autonomous.
So, the intermediate, and arguably more optimal solution is to involve humans in the loop.
This is where Virtual Reality comes in as it allow users to intuitively interact with
robot systems.
Additionally, in recent years, hardware and software development for Virtual Reality (VR)
has enabled researchers to leverage representations of the real world in a virtual environment.
However, when rendering data information in VR, several research challenges raised.
This include what data to show users, or how to effectively represent and transfer information-rich
and dense data sets to a VR environment in real-time; and what influence data limitations
have on a users ability to perform manipulation tasks
In this paper, we conducted a user study to determine the ideal configuration for the
VR-based control interface.
The user study investigate the effect of real-time sensor data variances on humans performing
straightforward assembly tasks in VR.
Experimental results were collected from 12 participants performing a total of 95 tasks.
The tasks required users to manipulate objects using streamed sensor data, such as point
clouds and images in a VR environment.
A multi-computer setup was devised to acquire, transmit and render real-time RGB-D sensor
data into a VR environment.
The data variances interested include the preferred visualisation modality, frame rate,
colour information, and resolution.
The system consists of two machines connected via Ethernet to minimise latency.
One machine obtains data from a RealSense camera via the ROS environment.
The raw depth and colour images received from the camera are compressed before being sent
through the network.
The second machine receives and processes the sensor data, then renders it into the
VR environment.
As mentioned before, the data is transferred using compressed images.
This is to enable real-time data transmission between the two computers.
Even though an existing ROS package is used to manipulate the image compression process,
an extra step is required before compressing the raw images.
The depth images received from the cameras are encoded in a 16-bit format, which is unconventional
for general image compression.
Consequently, two image streams are constructed from the original depth image stream: an upper
and a lower 8-bit image.
The single colour and two depth images are then compressed using JPEG, with the highest
compression quality to minimise data loss and then transferred to the second computer.
After the three images are received on the second computer, the software decompresses
the images, reconstructs and renders the point cloud in the VR environment.
The point cloud reconstruction is implemented using a pinhole camera model to convert each
pixel of the depth image into a 3D point.
By processing the depth image streams, a mono-coloured point cloud is obtained.
This is one of the visualisation modalities that is presented to the users during the
experiment.
The RGB image stream can then be used to construct the colour point cloud by associating each
point of the mono-coloured point cloud with a corresponding pixel in the RGB-D image.
An experiment was designed to investigate the effect of visualisation parameters on
the participants' performance and VR experience.
The experiment involved the assembly of a puzzle consisting of five parts whilst immersed
in a VR environment.
The assembly task was chosen to be straightforward to minimise the advantage from personal experience.
Fig. 1 on the left is an example start setting of the unassembled puzzle pieces, and Fig.
2 on the right is the puzzle assembled[TL1].
Each participant was asked to perform the assembly task 9 times, each with different
puzzle starting positions and a randomised visualisation setting.
Within the user's VR scene, the pieces of the puzzle are stationary on a static surface.
Both the camera and the VR headset position are localised based on the puzzle's position.
In the first 3 settings, the image stream with both coloured and depth image was displayed
to the user.
Each of these settings has a different image resolution and framerate.
The other 6 settings visualise data as point clouds.
Among these, 3 settings present mono-coloured point cloud and the other 3 present coloured
point cloud, all with varied density and framerate.
After each trial, the researcher recorded the participants completion time and requested
the participant to provide responses to record and evaluate their subjective cognitive workload.
The three statements which were utilized to evaluate the participants subjective cognitive
workload are presented on screen.
Fig. 3a shows the completion time of the participants.
Fig. 3b, c and d, which is the second to fourth box plot from left to right, shows the responses
from the participants on the Likert Scale towards the three statements.
For all figures, In Settings 1-3, colour and depth images were displayed to the participants.
In Settings 4-6, mono-coloured point clouds were displayed, and in settings 7-9, coloured
point clouds were displayed .
The experimental results indicate that RGB and depth image streams are the preferred
data visualisation modalities.
However, there are several factors that may confound the findings of the experiment.
First of all, since only one camera was used during the experiment, the reconstructed point
cloud could easily be occluded by the participant's hand and surrounding objects.
Secondly, since the assembly task was straightforward and required little depth perception, the
effect of the supplementary information provided by a point cloud is not significant.
Furthermore, the point cloud frame rate is limited due to hardware limitations.
Therefore, the image stream with a high frame rate and resolution is preferred by the user.
On the other hand, observations of participant performance and responses showed that colour
was the perceptual enabler for point cloud data.
Mono-coloured point clouds were reported to have little to no comprehensible visual feedback.
Therefore, in most settings that visualise point cloud, participants reported that they
are more dependent on tactile feedback than visual feedback.
This can be seen in the Settings 4 5 and 6 of figures 3b. in which the lower score indicates
higher dependent on tactile feedback
In conclusion, this paper investigates the effect of variances in real-time VR-based
sensor data visualization on user performance in an assembly task.
The variations include visualization modality, sensor resolution, data compression, and frame
rate.
In addition to RGB and depth image streams, the VR-based system was shown to transfer
color point clouds in real-time.
Future work is needed to investigate remote robot control tasks that require participants
to solely rely on visual feedback.
Other visualization modalities, such as meshes that could help both reduce the sparsity of
the data and produce a more intuitive representation for VR-based control operations will also
be considered.