from the mixed reality engineering team to take an early look at Microsoft Mesh a collaborative mixed reality platform from Microsoft that brings multiple participants into a 3D space to collaborate.
So Simon thanks for joining us today to introduce Microsoft Mesh.
thanks Jeremy – it’s great to be on the show.
So, Microsoft Mesh is one of the biggest announcements this year. Can you explain what it is?
Absolutely. One of the easiest ways to think about it is that Microsoft Mesh connects the physical and the digital worlds
allowing us to transcend the traditional boundaries of space and time.
You feel true presence, you can create together from any device of your choice, whether that’s AR, VR, PC or Phones.
For our developers, Microsoft Mesh can build immersive, multiuser, cross-platform Mixed Reality applications
leveraging AI-powered tools for Spatial Maps , Immersive Presence, Holographic Rendering , and Multi-user Sync
And you can leverage Microsoft Mesh to conduct virtual design sessions, learn together virtually, enhance virtual meetings,
host virtual social gatherings and meet-ups and much more with a geographically diverse set of participants
So now you can more easily interact with others as if you are face to face and collaborate with 2D and 3D content.
That’s pretty significant, this now expands the horizon for who gets to benefit from this tech. So as a user, how do I connect to Microsoft Mesh?
We just released a preview experience of Microsoft Mesh for HoloLens 2 – this is the first environment that leverages the Microsoft Mesh platform to bring people together.
And you’ll soon see a number of apps that take advantage of the Microsoft Mesh platform both from us and our ecosystem.
And the beauty of this is that you can join the experience from anywhere. I know earlier you recorded yourself Meshing with your team.
Why don’t we look at that and from there we can explain the tech behind it?
Absolutely, I‘m excited to see what you think....
It's so wonderful to be here and finally able to take the wraps off Microsoft Mesh
And being able to give a tour to you all. Before we get started.
Let me give you a quick tour of the setup. What you are seeing
Is a custom camera rig that gives you the perspective as if you are wearing a HoloLens
With that, let me flip my HoloLens visor down and enter Microsoft Mesh
Foundational to Microsoft Mesh is this notion of spaces - think of spaces like
a conference room, except it's something that is persisted across time and geographies
I have my home space, I have my pipeline review space
where my colleagues are waiting, so let me go ahead
And enter the pipeline review space
I can see here my dear friend Nishant
waiting in the review space, so Nishant
Great to see you here. Hey Simon, how are you doing?
Hello - hey Nishant, can you explain a little bit of what you are seeing?
Absolutely Simon, I see an avatar of yours and I see your avatar right across the room from me.
I see this awesome hologram of a table inviting me to collaborate
Which is inviting me to bring in 3D models to work and collaborate on them.
And of course as I lift my hand I see the hand menu where I can bring in more people
into the space, I can go ahead and change my avatar and appearance in the experience.
I can annotate, draw and bring in more shapes and figures into the space. I can also bring in 2D and 3D content to collaborate on.
Hey Nishant, since we are here to take a look at the project that Dorrene worked on
Do you think you can quickly invite her and see if she can join us in the space?
Certainly Simon, let me invite Dorrene
While Nishant is doing this I wanted to mention that Microsoft Mesh integrated with Azure AD (AAD) and Microsoft Accounts (MSA)
So that hosts can invite others into shared spaces in a simple and trustworthy manner.
When I lower down my visor
I can see that Nishant has already invited Dorrene into the space. Hey Dorrene!
Great to see you here. Hi Simon! Hi Nishant
Dorrene so great to see you. Do you recall the pipeline project that you worked on?
It needs a quick review - if you could kindly bring in that model from your OneDrive
so that we can take a look at it and make a few recommendations to our engineers, that would be great.
Absolutely Simon! Let me pull it up from my OneDrive. What's great about Mesh
Is the integration with the Microsoft Graph
What we show here is actually a model that our customers use today.
This is a part that is used to prevent spills
But the idea is that a wide range of data can be shown inside of Mesh.
Your connections, your collaborative efforts, your workflows and even your documents
transfer very well from a 2D space into a 3D space
So one of the things about collaborating in mixed reality is the 3 dimensional fidelity
That it offers and while each of us can have a unique perspective.
We all can rotate, translate, pan, zoom add or redact from this model
To bring clarity to decision making
while reducing the cost and complexity of being in in-person meetings
One of the things I miss from in-person meetings is the ability to annotate.
On a whiteboard or on paper.
And with Microsoft Mesh enabled applications
I can bring up my hand menu
Bring up my annotations here
And my fingers turn into essentially a pen
Through which I can annotate in the space
I can see here that while I've been doing this, my colleagues have annotated here as well
Yeah, and speaking of annotations, I think there is something here we should look at
Can you come over here and take a look quickly?
Sure
Great I'm actually noticing that this space on the model here
ay actually need a further look from our engineering team
It seems like there might be something out of date
Do you think they can take a look at thing maybe early next week?
Yeah I can request the engineering teams to come and take a look
One of the benefits of collaboration using Microsoft Mesh
Is the ability to persist these spaces across time and geographies.
So the engineering team can come and join the space
And pick up precisely where the team left off with the annotations
With the 3D models and with the point of view that
Dorrene wanted us to see.
And at any time if they need the help of Dorrene
or Nishant for expert eyes or hands
They just know that they are just a Mesh tap away.
So thank you Nishant and Dorrene for joining us on short notice.
We look forward to working with you again on another project soon!
With that I've given you a quick tour of Microsoft Mesh experience for HoloLens.
Awesome stuff, so that was the preview experience of Microsoft Mesh with HoloLens,
it almost felt like I was in the room watching this from the 3rd person perspective.
So, Simon, let’s unpack this a bit further, as under the covers, I know we are simplifying a lot of complexity to make this experience possible.
Right, it’s taken years of work to be able to virtually bring people together. We could do it 3 years back with unbounded compute and bandwidth
but with Microsoft Mesh we have cost and complexity reduced the AI to fit in 7 watt passively cooled head mounted devices.
Each of those AI capabilities from Spatial Maps to Presence to Rendering and Sync is decades of technical investment
so that as a developer you can focus on solving business problems while the Mesh platform solves the hard technical issues.
Importantly, with Microsoft Mesh, privacy and security are built into the platform, from integration of our identity services and encryption across all data at rest and in transit
Also, as you saw while I was hosting the Space from my physical environment, participants had a neutral view focused on shared content and the privacy of my personal space was protected.
And what we showed is just one example of the types of mixed reality experiences that you can expect.
Right, in fact the Microsoft Mesh platform can do even more. Can you dive deeper into the underlying platform capabilities?
Absolutely, so beyond what we just showed, Microsoft Mesh gives you a robust developer platform for building collaborative mixed reality experiences.
At the Core Platform – as I mentioned before Microsoft Mesh is integrated with: Identity with options for Azure Active Directory and Microsoft Accounts
It uses the Microsoft Graph to make it easy to intuitively connect users, content, and activities
This is all running on Azure and you don’t need to worry about provisioning backend services like live state sync or compute to run apps like these
Then as I mentioned we give you all the building blocks to light up collaborative mixed reality experiences with Spatial Maps; 3D Presence; Rendering and Multi-user sync. And as a developer, you can access all of this using our SDK.
So it’s pretty comprehensive, can we take a closer look at the capabilities, starting with Spatial Maps.
So with spatial maps, you can share and persist holographic content. This is what makes holograms stay in place across time, space and devices.
With Mesh, those local caches are merged and optimized to have a global understanding of the environment they are in.
And this global trellis enables content to be anchored, device points views to be shared and 3D models to be collaborated on.
How were you able to bring everyone into the same space.
That’s where ‘Presence’ comes in Presence makes it possible to represent ourselves in a more immersive way in the digital medium.
The fundamental parameters of digital presence are realism and reach. Microsoft mesh covers both ends of this spectrum.
For example, on the one end of the spectrum Mesh delivers the most accessible 3D presence delivering representative avatars with inside out sensors already on the devices that you carry.
The mesh platform is adaptable enough for you to bring in your own avatars and as you saw we provide a set of out of the box avatars too,
And on the other end of the spectrum, Mesh delivers the most photorealistic 360 holoportation with outside in sensors.
And of course, the holy grail is to merge those worlds and that is for the future.
Presence also brings spatial audio so even as you move in the space, the audio will be directionally emitted in the 3D space consistent with your relative position.
It’s truly magical and you have to experience it to understand the connection and engagement it delivers to your apps.
Of course, beyond the movements of the avatar, you are also able to bring in models that can be manipulated in the space.
Yes, that’s where Holographic rendering comes in This allows you to bring in 3D models and provides the rendering power that you need when you need it
You can do local rendering optimized for power and thermal sensitive devices and with Microsoft Mesh you can also leverage
the power of the Cloud to bring in the compute required to render more complex models that are very commonly used in industrial settings.
Here you can see an example of the difference in realism between a 100,000 and 100,000,000 polygons
This helps you supercharge your apps with high-quality interactive 3D content where every detail matters
So, the Spatial Map capability helps us to know the space we are in; we can bring participants and content to the space through presence and rendering.
Now, one aspect that caught my attention in the demo was that all participants had a common perspective of the model and each other. What made that possible?
This shared context is enabled by the multiuser sync capability, which is the magic that brings the entire experience together. It’s the TCP/IP of Mixed Reality collaboration.
This is what lights up any pose updates from participants and any holographic transforms that are happening in the space
as you can see here with our engineers testing this out using markers and passing them between each other – notice the live action picture and picture view
is in sync with the shared space at the bottom. And these are moving synchronously, even though the participants are halfway across the globe
And all of this happens with under 100 milliseconds of latency because of the compression technology built into the platform
How can you harness this as a developer?
We’re building a SDKs so that you can create your own immersive mixed reality apps The SDK will
support a diverse set of tooling across popular middleware.
We have also deeply invested in a UX toolkit for Mesh.
The goal is to provide several common UX constructs that you can pick and choose from so that you can build professional looking apps.
To get started or sign up for the preview, where can people find out more?
The Microsoft Mesh experience for HoloLens is in preview today. And You can try it out for yourself by signing up at Microsoft.com/Mesh
If you are a developer, you’ll find more guidance at aka.ms/meshdocs
And to see the developer experience in more detail, check aka.ms/MeshIgnite
Thanks Simon. And for everyone watching, we’re looking forward to seeing what you build on Microsoft Mesh.
Of course, keep watching Microsoft Mechanics for the latest updates, subscribe if you haven’t yet and we’ll see you soon!