(mechanical whirring)
- Coming up on this special edition
of Microsoft Mechanics
we meet New York-based real estate company, RXR Realty
to learn about how they're implementing IoT and AI
on the edge with Azure to return people
back to the office safely during COVID-19.
So I'm joined today by Cory Clarke,
and he's in charge of digital products at RXR,
joining us from their flagship building
in 75 Rockefeller Plaza in New York City.
Welcome to the show.
- Thanks for having me. It's great to be on the show.
- And thanks so much for joining us today, Cory.
By the way, for those of you who are new to RXR Realty,
they're one of the largest real estate companies in New York
with over 27 million square feet of real estate.
Now in normal times, around 75,000 office workers
occupy their building spaces.
And while New York was the epicenter of COVID-19 early on
its positive recovery on many levels
provides an interesting model
for how to safely go back to normal life
in the middle of a pandemic.
Now RXR is offering its own unique perspective
by implementing its RxWell platform,
an IoT and AI-based solution that supports safe return
of people back to the office
and this arms employees with the right insights
and helps the company and their tenants
and property managers to monitor the health
of their buildings and also apply the right strategies.
Okay, so Cory you're tackling
one of the toughest challenges then
that every employee or organization right now can relate to,
which is getting people safely back into the workplace.
So where do you even start?
- Fortunately for us, pre COVID-19, in January,
we started to implement a strategy
to transform our traditional real estate footprint
into an intelligent network of digitally connected spaces
using technology, IoT sensors in the cloud.
So when COVID hit in March,
we were able to immediately start pivoting,
focus on wellness and safety
and begin to design the RxWell platform.
And by June, in as little as three months,
we were able to get everyone back to work.
We had hardware pilots up and running in three buildings.
The first version of our software
was in all of our buildings.
Since then we've rolled out a combined hardware
and full software solution to the rest of our portfolio.
- Right.
And it's pretty staggering just how fast
you were able to re-pivot your existing plans
that you had in place,
not just to operationalize CDC guidelines
for public health, but also figure out the best way
of getting everything aligned and really
to create a safe environment.
So how did he go about doing that?
- So once we had our new plan it was really just a question
of bringing together the right people.
Fortunately, we had a team of almost 30
across RXR, our technology partners, Microsoft,
to ensure we could deliver on that.
And then our approach included
a technology selection process
to put enterprise-level security as the primary driver,
as well as an overall design process
that include various users and customers.
- So can you walk us through then
some of the user experiences?
- Sure.
As you mentioned at the beginning,
we have three target user groups,
the first being office workers
who need to feel safe in their return back to the office.
So let me show you their mobile app experience.
So as an employee, you can download the app in your phone,
access it by the web.
We designed it to help you make informed decisions.
Because of the state mandated target of 50% space occupancy,
we've implemented a tool for tenants
to manage their teams' workplace rotation schedules.
Then as an employee, you can check your rotation slot,
see when you can come into work.
And then to enter the building,
you need to get the daily access pass,
which you obtain by completing
a simple COVID questionnaire
that asks you if you've tested positive,
if you've experienced symptoms,
if you've come in contact with someone with COVID
in the past 14 days, or if you're required to quarantine.
And then when you successfully complete that
it gives you a pass for building entry.
It's valid for the day.
So before you make the trip to work,
you can also review the overall building wellness pulse
with metrics like air quality, it's overall density,
as well as data about the local community.
And if you're using our wearable
for personal social distancing,
you can view your personal social distance metrics.
It shows you the number of people
you've been in contact with each day and for how long,
as well as trends over time
and how your behavior compares with others in your company.
This helps improve your overall social distancing behavior.
And we help you further by getting information
to plan when to enter the workplace
by reporting foot traffic in real time,
those peak times throughout the day to avoid crowds.
And lastly, you can check the time
your space was last cleaned.
And then for any area you can request cleaning on demand,
and see that updated in the app.
- Okay. So that's the employee experience then?
How do property managers then interact with the platform?
- So the property manager has the most comprehensive view
of building data in real time,
so they can dynamically adjust policies and procedures,
get the best level of service to people in our buildings.
We took an approach that gives an oversight
of all the COVID-19 public health compliance,
but in a way that respects individual privacy.
So as a property manager,
you can view all the buildings in your portfolio,
key metrics such as whether they're managing
to New York state's target capacity at 50%.
So for every building,
you can monitor overall building health based on an index.
We've designed that to take in account
things like reported health issues, temperature scans,
overall building density and air quality metrics.
And you can also drill in to explore further.
For example, with COVID, air quality is super important.
So we know fresh air reduces risk.
So we exchange all the air in our buildings
with outside air every two hours,
we show the level of CO2 in the air
as an indicator of how fresh it is.
We also know that to reduce the risk of COVID transmission,
we should keep humidity levels to around 40-60%.
So we show that factor in our health index.
And then you can drill into a sensor
and see air quality on a specific floor
and make sure it's within optimal range.
Under occupant health, you can quickly see if tenants
are experiencing an outbreak with data on reported cases,
general absenteeism due to illness,
as well as trends in temperature scans.
And then you can look specifically at community metrics
like COVID hospital rates in neighborhood.
And then also, as you know,
one of the hardest things with COVID is ensuring compliance
to public health measures.
And we're constantly monitoring social distancing
in a de-identified way throughout public areas.
- So what about your third audience then?
The companies, maybe there are facility managers
that are actually tenants in the spaces.
What are they then able to see?
- So the tenants, they can see a subset
of what the property manager sees.
You'll see occupant health with case count,
today's high temperature scans,
temperature scan trends over time,
building access metrics, absentee illness tracking,
and hospital admission information.
And it helps you get a pulse on your capacity
by reporting how much your workforce has returned,
or if more needs to be done to improve compliance
to public health measures.
- And what I really love about these tailored experiences
is that they're transparent and also data driven
and make it easy for everyone to make informed decisions
around their actions.
But what's the tech then behind the scenes
that we don't see here?
- So these core user experiences
are fueled by the intelligence that we place at the edge.
It connects that back up to Azure.
We've architected a three-tiered model.
We have specialist IoT devices reporting data
we've placed throughout our buildings.
We collect data from existing
or specialist systems on premise,
and then we've installed an Azure Stack Edge device
for local inferencing of security footage.
The data flows to Azure and all the data ends up residing
in a single Azure SQL database.
At the same time, we have a cold path
where the data is stored after the after-the-fact analysis
of data like building energy consumption.
And in the future, the goal is
to apply more machine learning,
drive insights and building automation.
- So I would imagine you have a lot of IoT sensors.
Where does the placement of those IoT sensors go?
- So our IoT footprint differs by building,
but if we take our flagship building here as an example,
so we have air quality sensors that we've placed
throughout the building to measure inside air.
And then in our fresh air dampers or intake ducts,
we measure outside air.
This data is processed with Azure Stream analytics
to monitor CO2 differentials
between the inside and the outside air
to give us an accurate read on fresh air circulation.
Then we have two common types of sensors for occupancy.
We have a people counter
that counts people entering and leaving a space.
And for example, we've placed those in the lobby turnstiles.
And then we also have binary occupancy sensors
that we placed in other communal areas.
So these sensors connect to Azure IoT Hub via SaaS Gateway.
And then we use Azure Stream Analytics to process the data.
We have an Azure function that listens for updates,
which are sent to an Azure SQL database.
Then we apply our own data model
that queries against the KPIs,
outputs the results into the RxWell apps.
In this case, it helps us report
on building traffic and occupancy.
- Okay, so how fresh or real time then
is that data coming in?
- It's nearly real time which is a really big reason
for using Azure Stream Analytics.
Let me give you an example.
So we have this clean button that cleaners press
to confirm sanitation of a bathroom.
Once the button's activated, Azure Stream Analytics
caches this information in memory.
At the same time, Bluetooth sensors detect
if there's a cleaner with a badge in the vicinity.
Within seconds of the clean button being pushed,
Azure Stream Analytics checks to see
if that badge has been located.
And once the presence of the badge is confirmed,
an Azure function pushes the status update
to the external digital signage,
the date and time the bathroom is cleaned.
- What I like about this is you've accounted
for then false positives in your solution
that rules out maybe the possibility
of someone being a little sneaky,
other than the cleaner in pressing that button
to try to confirm cleaning.
But how much intelligence then is kind of baked
into the Edge devices themselves?
So this can vary throughout the implementation,
but for the most part, these are dumb devices,
other than the fact that they collect and transmit data.
So for example, the displays outside our bathroom,
these are low-powered, battery-operated, Bluetooth devices.
They don't need any wiring which makes them really easy
to mount anywhere, we can just use stickers.
- I know a lot of people watching are probably wondering
about the security, given that a lot of times
these IoT devices and Edge devices are web facing.
So how are you handling the security there?
- Well I think for anyone who's new to enterprise IoT,
this isn't a raspberry pie, DIY kit.
Security's at the crux of the solution.
Our IoT devices communicate through secure channels,
authenticate using security tokens.
So this is achieved by deploying our Azure IoT SDK
into our SaaS device gateways.
We also use Azure Key Vault
to assign cryptographic key pairs to those gateways.
If we take our bathroom configuration, for example,
data from the button and badge sensors
is communicated via the gateway.
And the Azure function that triggers the message
on the display is also running the IoT SDK
for secure token-based exchange of data.
- So it's good to hear that security
is basically inherently baked in,
but you mentioned that you can continue leveraging
on-premises systems for your building metrics, for example.
How does that work?
- So we have, for example,
all our premises have a building access control system.
It verifies and logs employees
as they badge in and enter the building.
And this gives us a second data point around occupancy.
It provides more granular insights
such as where the floors are, where each person's heading.
And this is correlated with data from our people counter.
In the lobby we use it to accurately assess
the building and floor occupancy
against our target threshold of 50%.
And it's important to note again,
this is completely de-identified
and it's passed through the same flow
as our Azure SQL database for reporting
to the RxWell platform.
- Right, and one thing I noticed is that you were able
to report on the health of people that are in your buildings
in both the employee app and the property manager app.
How were able to do that?
- So there's three ways we do that.
So we've pull data from the COVID self attestation,
that's in the employee app that we showed you earlier.
Then we have tenants reporting
if there's any positive COVID cases in the building.
And then finally we have these thermal cameras
that we've installed in the lobby
that monitor people's temperature.
And then we have a logic app that adjusts the data
and sends it to Azure IoT Hub.
So if person has an elevated temperature,
that's a particular event.
We record that in Azure SQL database for our reporting.
And we also track the average temperature
for all the individuals entering the building
and use that trend as a potential indicator of outbreaks.
- Right. This is fascinating stuff.
Can you explain then how you're using though
Azure Stack Edge Devices?
- So we deployed the Azure Stack Edge Devices with GPU
to do inferencing on our security footage.
This has a container
that is running a prebuilt Azure computer vision model
called Spatial Analysis to analyze
the data stream of video footage.
So here's the output of the analysis,
and this is our lobby space and you see the foot traffic.
And then from spatial analysis,
we're using two distinct models.
People counting,
that allows us to count people in a designated area,
and then social distancing,
which allows us to observe compliance to social distancing.
Again, one of the most important aspects
of this implementation is that all the data reported back
is de-identified.
So we can look at our social distancing compliance
and aggregate, determine strategies
for improving compliance,
such as how we route people to the lobby.
- Right, and just to explain,
the Azure Spatial Analysis models,
they're part of Azure Cognitive Services,
and they really comprise high-quality models
that are built by Microsoft's own data scientists.
The people counting, and also the social distancing
are in preview today.
And you can find out more about those
at aka.ms/SpatialAnalysis.
And what makes these models high quality
is that Microsoft has trained them
to a super high level of accuracy
to work in any environment
and they run on the edge with data control,
privacy and scale built in by design.
- And to the point, what's awesome is
these models are pre-trained.
We literally didn't have to do anything. They just worked.
So here, I've connected my Visual Studio code editor
to Azure IoT Hub by an extension.
So I can view the YAML configuration file.
So this is the configuration
for special analysis, personal distance.
Right now it's configured for a single camera feed.
The Azure Stack Edge happens to have two GPUs,
but if I had multiple cameras,
here's where I could assign a GPU
to each camera to run the model.
You'll see the minimum distance threshold
is set to six feet.
And I can modify that if I want to.
Next, you can see analysis output.
Here's where we see the person counts,
which is the number of people within the frame.
And we can see here,
it's detected that two people are too close.
We're able to take that output and record the event
via Azure Stream Analytics and curate the results
as part of our social distance compliance reporting.
- Okay, so with all these different IoT devices,
how have you approached then,
the management of those devices?
- That's the core advantage of Azure IoT Hub
because it's end-to-end device management,
massive scale, seamless integration
with other Azure services.
So here I'm in Azure IoT Hub.
As you can see in the overview blade,
the volume messages is around 300,000
from 50 connected devices right now.
But that is just a portion of our portfolio occupancies,
only around 10-20%.
So as we increase building and occupants,
we can scale elastically to meet our needs.
And scrolling down, I can get a single view
over all our IoT devices.
We register and authenticate all devices
before they communicate with the IoT hub.
I can also see our Device Gateways
as well as our Azure Stack Edge
which is provisioned like any other IoT device.
And we can control the modules that run on it.
Next, tools like IoT Explorer allow us
to see live telemetry devices, validate device health
or troubleshoot issues.
So I have end-to-end device management,
including configuration control for devices.
If I go back to our IoT hub,
you can see we're able to do bi-directional communication.
So we can send messages to devices,
execute direct methods and control device configurations
to things like device twins.
- Right, and this is probably one of the best examples
that I've seen of IoT and AI on the edge,
but what's next then on the horizon for RXR?
- All this data opens up a world of possibilities.
We're able to start providing
more personalization for users,
we are able to understand space usage
and employee productivity,
and we're able to optimize our operations of our space.
And we're looking to move this to Azure Digital Twins,
make the data easier to model and analyze.
So we're also working on improving the wellness offering.
We've partnered with the healthcare provider
to develop a new method of environmental testing
that uses special PPE for our cleaners.
This allows us to identify where in the building
there might be traces of COVID
so we can identify potential asymptomatic carriers.
Then we're also looking into options
for identifying face mask usage via machine learning.
And finally, with such a scalable backbone as Azure IoT,
we're able to offer this solution
to other landlords and occupiers.
You can actually get our solution in the Azure Marketplace
at aka.ms/RxWellSolution.
- Great stuff.
Thank you so much, Cory, for joining us today
and really sharing how you're tackling an issue
that really resonates with a lot of us,
whether we're an employee or an organization
trying to figure out how to operate during the pandemic
while keeping people safe.
Now, if you want to learn more about RXR's example,
check out aka.ms/RXWell.
And to learn more about Azure IoT and AI,
check out azure.com.
Hopefully we answered all your questions today
and also keep watching Microsoft Mechanics
for latest tech updates.
Subscribe if you haven't already
and we'll see you next time.
(dramatic mechanical music)