The metaverse is an ever-present spatial internet complete with personalized digital experiences
that span the physical and virtual worlds where everything and everyone can communicate
and interact seamlessly. Qualcomm Technologies is enabling the metaverse with cutting-edge research
and industry-leading products. In this video, you will see how our advanced
wireless research is enabling boundless AR, cloud gaming, and new sensing capabilities.
At Qualcomm Technologies, we are focusing on research and commercialization of new 5G use
cases such as oundless AR. Boundless AR uses split rendering to distribute the computation
between the server and the device to deliver truly immersive and enhanced AR experiences. While the
long-term vision to the metaverse is a 5G-powered AR glasses that connect directly to the cloud,
a near-term solution are Wi-Fi powered AR glasses that communicate to the cloud via a 5G-enabled
phone or laptop. In this demo, we showcase our optimizations on a prototype boundless AR system
where the AR glasses communicate with the phone over Wi-Fi and the phone communicates with the
server over 5G. The pose information and video from the user flows from the glasses to the phone
to the server. The server processes the data and sends back the encoded graphics via the phone to
be displayed onto the glasses. The round-trip time for this entire process is called motion-to-render-
photon latency, or M2R2P. The 5G and Wi-Fi round-trip times are key components of M2R2P. High
5G and Wi-Fi round-trip times due to challenging radio conditions can lead to poor user experience.
Our test system runs emulated boundless AR traffic over a 5G sub-6 gigahertz test network and Wi-Fi
6. In this example an AR user is at the far cell location with the phone in their back pocket. The
AR glass is undergoing yaw and pitch motion as the user looks around to observe the surroundings.
Our optimizations help reduce the 5G and Wi-Fi round-trip times by more than 28
percent on average and more than 38 percent of the tail, providing an improved user experience.
Additional gains are expected in more challenging pedestrian and vehicular mobility scenarios.
With our system level optimizations we are enabling truly immersive boundless
AR experiences over Wi-Fi and 5G. This is just the start. We are continuously innovating to
enable new and improved AR experiences. We are making the metaverse a reality.
5G cloud gaming uses computing power from cloud servers to deliver rich interactive
digital experience to users anywhere over 5G, a key component to enabling the metaverse.
For good user experience it is important to minimize the latency from user input
to on-screen action. This is also referred to as controller-to-render-to-photon latency, or C2R2P.
5G round-trip time is a critical component of C2R2P. A high 5G round-trip time due to challenging
radio conditions can lead to poor user experience. We now show the latency performance of a 5G device
inside a moving vehicle in our outdoor 5G sub-6 gigahertz system. We can see that
the baseline 5G system with our own device optimizations see significant latency spikes.
Our on-device optimizations significantly reduce the latency spikes resulting in responsive and
smooth gaming experience. Overall, our on-device optimizations can be seen to achieve 5G round
trip time tail latency improvements of 32 percent. Gains can be larger in more challenging network
and RF scenarios. Another key issue that impacts the user experience today is that applications are
unable to rapidly adapt bit rate and resolution to the changing cellular network and RF environment.
We have developed a 5G AI API for gaming, which helps applications dynamically adapt
to the radio and network conditions to deliver the best user experience.
We can clearly see that with our API the gameplay no longer starts when the radio condition changes
quickly. To summarize, with our optimizations gamers experience a smooth video with better image
quality and lower frame drop rate in addition to responsive gameplay. We are continuously
improving our on-device features to deliver rich interactive digital experience for the metaverse.
Understanding the physical world is a key requirement for the metaverse and at Qualcomm
Technologies, we are leveraging our wireless expertise and experience to help accomplish
that. In this demonstration we will show how 5G millimeter wave, deployed for wireless communications,
can be utilized for object sensing. Please note
the object that has been detected is device free with no ability to communicate over 5G.
It could be a person, wall, or even a large rock that is in our physical environment.
Here is a system simulation environment where we have one gNodeB and three remote radio heads
operating in 28-gigahertz band. We also have a group of millimeter wave devices that are scattered
in the test area. We call this cooperative sensing as both network and device are working together
to determine the location of the device-free object. The system utilizes both 5G positioning
techniques like a downlink angle of departure or time difference of arrival to determine
the location of the 5G device, and RF sensing to determine the location of the device-free object.
In a simulation, we have an object placed on the grass lawn, and through cooperative sensing we're able
to sense the surface of the object as indicated by the blue dots. Combining the dots allowed us
to reconstruct the shape of the object. We can also see the average distance error to object surface,
denoting the accuracy performance of our 5G RF sensing simulation and with more RRH participating
in cooperative sensing, the better the sensing accuracy. In conclusion, 5G not only delivers
wireless communication but also provides a better understanding of the physical world to seamlessly
merging with the digital and the virtual worlds, making the metaverse a reality in our future.
As we saw in previous demonstrations, the role of wireless and enabling the metaverse
expands beyond data communication. That is why we are evolving the connected intelligent edge,
powered by 5G, edge cloud, and on-device AI to optimize the performance of secure non-data
services. They can bring enhanced metaverse experiences, and some example areas include
precise positioning, relative ranging, timing resiliency, and a few others. Many of this can
be enabled with 5G as part of the ongoing 5G technology evolution. To support the services,
there are multiple system architecture options. First with a device only model the device can
independently support the service without interacting with the network. Another model
is for the device and the network to jointly collect and process data to support the service.
In this case, either the network or the device can take the leading role, while the other assists. An
example is 5G precise positioning, where the device can measure the positioning reference signals
and the computation can be done on the device or in the network. To optimize the system for
non-data services we can develop new interfaces or APIs across the cloud, network, and device
platforms. This can facilitate efficient horizontal scaling, providing new entrance with fair, level-
playing field with the larger vertical players. We believe a hybrid cloud and device platform can
offer the most flexibility for meeting diverse service requirements. In most cases, there is a
service orchestration trade-off that needs to be made. If service hosting is predominantly in
the cloud or the network, the on-device value that can be derived would be mostly from connectivity,
while the on-device value is mostly derived from compute when the service hosting is on device. In
conclusion, this is an area of our longer term research, and we're exploring ways to design an
enhanced and flexible platform that allows dynamic model selection that optimizes for the use case.