>> Thank you so much for joining today.
My name's Cameron Micka, and today I'm going
to be talking about how you can get
started developing for HoloLens 2 using the Unreal Engine.
Before jumping in, I wanted to give a brief introduction.
If you're curious to learn more about me,
my name is again, Cameron Micka.
I'm a senior software engineer on the Mixed Reality Design Team.
I've been at Microsoft since 2016.
Before that, I mostly worked in the video game development world.
At Microsoft, my focus has been helping
our partners and independent software vendors
with mixed Reality App Development.
I've also worked a lot on the Mixed Reality Toolkit for Unity,
where I'm the author of The Standard Shader system.
But recently, my focus has been
on the Mixed Reality Toolkit for Unreal,
as well as UX tools.
Next, I'm going to quickly walk you through\
how you can start developing for HoloLens 2,
using the Unreal Engine.
I recommend installing Unreal Engine
through the epic games launcher,
just like you normally would for any platform.
Just take note to check the HoloLens 2 option
as one of the target platforms during installation.
To make sure your project runs well on HoloLens 2,
it's good to sanity check your project settings.
If you're creating a project from scratch,
make sure you configure your project to use
the scalable 3D or 2D settings as well
as mobile and tablet settings.
If you already have a project created,
make sure to sanity check the rendering settings as below.
I'll talk a bit more about this later in the presentation.
Once you've opened up your project,
there's a few plugins that come bundled
with Unreal Engine you'll need to enable.
The two plugins are the HoloLens plugin,
under augmented reality, and
then the Windows Mixed Reality plugin under virtual reality.
Once your project settings are
configured appropriately for HoloLens,
the last step is to start an AR session.
The easiest way to start an AR session is to call
the start AR session blueprint node from a begin play event.
Note that node also takes an AR session config,
which you will need to create.
That's it for the initial setup.
You're ready to start creating HoloLens 2 applications.
But before you get going,
I recommend everyone also
use the Mixed Reality Toolkit for Unreal.
Next I'm going to go over the Mixed Reality Toolkit for Unreal,
also known as MRTK-Unreal.
If you've never heard about MRTK-Unreal,
that's fine because it's relatively new.
MRTK-Unreal is a container around various Unreal Engine plugins.
Our very first plugin is named UX tools,
which as the name suggests,
helps our developers build out user experience elements.
During our Unreal HoloLens 2 beta,
having a toolkit similar to what was in unity,
was one of our top customer requests.
Building on our learnings from MRTK-Unity,
we're making MRTK-Unreal more modular,
which means developers available to use features and
systems they need without pulling and ones they do not.
This image on the right, you can think of MRTK-Unreal as
a blue container around various plugins,
UX Tools being our first,
with more plugins coming soon.
Next step, UX Tools.
Speaking a bit more about our UX Tools plugin,
it's hosted on GitHub as an open source project,
so you can navigate to
the blue URL right now and check out the code.
It contains a Unreal game plugin,
as well as an example project which
contains tons of examples to help get you started.
The plugin itself is a collection of C++ functionality,
blueprints, example levels, and tons of documentation.
One thing to note, we're currently not in a position
to accept external contributions via pull requests.
However, please feel free to
provide contributions and feedback in the shape of bug reports,
suggestions, and feature requests.
If you don't want to download and build
the UX Tools plugin from GitHub,
it's also available on the unreal marketplace.
Here's a brief overview of the UX Tools development timeline.
Back in February 2019,
Microsoft and Epic Games
announced that HoloLens 2 support was coming.
In August 2019, beta
support for that shipped in Unreal Engine 4.23.
In December 2019, we began working on
UX tools, and since then,
we've released a few updates in May,
August, and recently October.
At the same time,
Unreal Engine has been improving to support
HoloLens 2 with new features as well as performance improvements.
The goal of UX Tools is to provide components that
make building UX for Windows makes reality easy.
Next, I'm going to go over a few of those components.
First off, we have a HoloLens 2 style pressable button.
This button mimics the visuals and behaviors
you find in the HoloLens 2 operating system shell.
The button is highly configurable,
and it's based off the pressable button component
already found in UX Tools.
Building on top of the HoloLens 2 style button,
we have a few toggle button permutations.
We have the classic Toggle on, Toggle Off button.
We have a Switch button,
A Check button and a Radio button.
Any of these buttons can be used in groups
to ensure only one button is checked at the time,
and each button can be customized to get your own look and feel.
Speaking of customizing buttons,
one thing we did was surface all of
the commonly used customization properties to the top-level actor.
In this example, we've created a camera button.
There's a custom size, it's extra wide,
it has a special button icon,
and a unique label.
Specifying button icons can sometimes be a laborious task.
To make this easier, we made a custom editor window that
displays all the available icons in a font.
Our button icons come from
font files and are rendered just like normal text.
Another cam Control found in the UX Tools is the pinch slider.
A pinch slider can be thought of as a hologram that moves along
a rail and reports back a value between zero and one.
Here you can see a pinch slider is used to change the color on
hologram or animate the landing gear on a lunar lander.
Just like the pressable button,
the pinch slider is highly customizable,
and contains a few exposed properties in the Details panel,
to change the look and feel of the control.
The bounce control is a useful component that's used to
change the rotation or scale of a hologram.
Here you can see these two blue hands
grabbing the affordances to scale up the object.
These affordances mimic the visuals and behaviors that are
found in the HoloLens 2 operating system shell.
Often we want to keep a few buttons or
other controls within arm reach of the user.
This can be accomplished with Near Menus.
Near Menu is provide behaviors which
constrain the menu to the user's location,
and allow users to manually place the menu
by grabbing it or pin it to a location.
I've covered a few UX controls that are often used in
conjunction to create more complex menu setups.
The UX Tools contains a few examples and utility components
which demonstrate how to set up nested menus or menu hierarchies.
When interacting with holograms from afar,
you need some way to
indicate what you're pointing at or what you're grabbing.
That's where the HoloLens 2 style hand rays come into play.
When grabbing a hologram,
they bend to constrain to the hologram and
your hand and when pointing into empty space,
they provide a dashed line to give you
an indication of where your next hit target will be.
Constraining UI to the hand is one of
the more popular request from developers,
and UX tools contains a few components as well as
examples on how to properly attach UI to a hand,
as well as show and hide it based on
the hands orientation or activation status.
Unreal Engine contains a workflow called UMG to
build flat or 2D UI layouts.
If your application already uses
UMG where you prefer using UMG for creating UI,
luckily UX tools can be used to interact with any UMG component.
We have a few components which are still marked as experimental.
Experimental means the API might change
or the component isn't quite ready for production.
One of those components is called tool-tips.
Tool-tips can be used to annotate portions of a hologram.
The second experimental component
currently in UX tools is the scrollable list.
Scrollable lists allow users to scroll through
a continuous array of UI elements such as buttons.
Because HoloLens 2 is a mobile stereo headset,
performance should always be considered when developing an app.
UX tools contains a few shaders to
provide HoloLens 2 style visuals,
as well as material functions that can be
applied to generic meshes that
perform well on device and fit
within the devices fill rate budget.
If you download UX tools from the GitHub repository,
it will contain an Unreal game project
that contains many example levels.
Each of those example levels can be
accessed from a central hub level,
this screenshot is from our next UX tools release,
and gives a sneak peek at what that hub level looks
like with upcoming components and examples.
Because HoloLens 2 is a mobile stereo device,
performance should always be at the forefront of developers minds.
In the following slides, I'll go over a few performance tips that
we've encountered when creating examples such as Kippy's Escape.
The first thing every project with sanity check is
that the project settings are configured
to be performing on HoloLens.
If you open up your project settings,
scroll to the engine section, select "Rendering",
expand the VR section,
and enable both instance stereo and mobile multi-view.
Mobile HDR should also be unchecked for best results.
If you need occlusion calling for a detailed scene being rendered,
it's recommended that you enable
support software occlusion calling
in the engine rendering section.
This let's Unreal do that work on
the CPU and avoid GPU occlusion queries,
which can perform poorly on HoloLens 2.
If you don't need calling for application at all,
it's best to turn it off completely.
For more details and recommended settings,
please navigate to the link at the bottom of this slide.
Next step is to perform a course profiling of your application.
Every HoloLens developer should have the frame rate
counter on when developing their application.
Luckily, HoloLens 2 contains a built-in frame rate display.
To access it, open up the Windows Device Portal,
here's the IP address to connect
to when you're HoloLens is connected via USB,
then open up the performance section and under system performance,
check the display frame rate counter and
headset and optionally display the frame rate graph as well.
After enabling the frame rate display and you
find your application is running below your target frame rate,
try using Unreal stat unit command to glean more information.
Stat unit will help you narrow down if
your app is CPU or GPU bound,
on HoloLens it also display
the current polygon and draw call counts.
Note on a HoloLens,
stat rendering doesn't always render 100 percent correctly,
sometimes it renders twice in one I,
so you may have to squint a bit to read the numbers.
Here, I'm enabling the stat unit command
when the user says toggle stats.
Once you've narrowed down from
a course level while your application is slow,
you can start using finer grain profilers
to find performance bottlenecks.
Unreal has many great profiling tools built right into the engine,
which you can learn more about at the below link,
Unreal also supports RenderDoc if you need
to perform complex render state inspection.
RenderDoc is available as a plugin built right into the engine.
HoloLens applications are often fill rate bound,
fill rate refers to the number of pixels
that can be drawn per second by the GPU.
Fill rate optimizations are normally focused on reducing
the number of operations that need
to be computed for a final rendered pixel.
This includes reducing the number of objects to render process,
the number of operations per shader,
the number of GPU stages for a final pixel result
such as geometry shaders, post-processing effects.
The number of pixels to render, again,
your displayed resolution and many other factors.
We found that the operations being
calculated on our lit materials
on Kippy's Escape were rather high.
To reduce this, we marked all of our lit materials
as unlit and use the custom material function found
within UX tools named
MF_SimpleLit to perform simple Blinn-Phong lighting.
It's important to keep an eye on the number of
vertices your application is presenting.
More vertices mean more vertices
to the process and the vertex shader,
and more data to transfer.
Think of memory bandwidth or load times.
Normally you decimate or reduce your meshes
in a 3D content creation tool like Maya or 3ds Max.
But Unreal has tools built right
into the editor that automatically decimate a model.
Decimation is quick and it allows you to iterate
on the settings appropriate for your application.
It's a good idea to keep an eye on the number
of draw calls your application performs.
You can think of a draw call as
your app instructing the GPU to render something.
Unreal automatically batch similar objects together that
contained the same measure material into
one draw call using it's in sync.
If you have actors that are static that you think
could be merged together to reduce draw calls,
try using Unreal's merge actor tool to automatically
merge meshes and optionally materials together.
In this screenshot, we've merged a bunch of
stones that a level designer placed individually.
Note, actor merging is a destructive process,
so it's best to do towards the end of development.
Next, I'm going to talk about our MRTK-Unreal roadmap,
as well as any extra support you can find online.
We've already shipped dozens of features within
the last three releases of UX tools.
In our next release,
we plan to support Windows Mixed Reality immersive headsets,
as well as Oculus Quest.
We plan to further support OpenXR.
Then we'll have a few quality of life improvements, bug fixes,
and add a new text box feature as well as
graduate the tool-tips in scrollable object collections.
As mentioned earlier, MRTK-Unreal is a collection of plugins.
Here's a sneak peek of our next plugin called graphics tools.
Graphics tools will enable you to create
beautiful experiences on HoloLens 2
while staying within your performance budget.
For more information on general HoloLens 2 development and Unreal,
here's a summary of links to documentation,
repositories, tutorials and samples.
One sample I wanted to call out is Kippy's Escape.
Kippy's Escape is hosted on GitHub as an open source example,
as well as a free store download if you want to try it out,
and we have a postmortem on the making of
Kippy's Escape on docs.Microsoft.com.
Thank you so much for listening to this presentation.
If you have any additional questions
about MRTK-Unreal or Unreal in general,
please don't hesitate to reach out. Thank you.