Hi and welcome to another in the Arm “What Is” program series. In each episode, we dive into a
tech topic to give you insight and perspective into some of the hottest design trends today.
I’m Brian Fuller, editor-in-chief at Arm and today we’re going to find out what is endpoint AI?
And to help us with that, I want to introduce Chris Shore, Director Endpoint AI Ecosystem at
Arm. When he’s not helping us understand what is endpoint AI, Chris is playing the piano
and ringing church bells. I kid you not, but that’s a whole different story! Welcome Chris!
Welcome Brian! We'll do the piano and the church bells another day,
but I'm happy to go on about endpoint AI.
Splendid, let's dive right in!
So what is endpoint AI, Chris? Very simply, the business of implementing intelligent functionality
at the edge of the network. Slightly less simply, taking the smarts that you usually run on big
iron in the cloud and relocating them to the widgets at the sharp end, right at the edge.
Is there a difference between edge AI and endpoint AI or are they two names for the same thing?
That's tricky just for only the second question, Brian! Edge AI and endpoint AI are often used
interchangeably. For me, the "edge" is a bit of a nebulous term. It can encompass multiple
devices like routers, edge servers, hubs, and the functionality distributed across all the devices.
For my purposes, an endpoint is where the network hits the real world,
where the analog meets the digital if you like, where the rubber hits the road.
So we're talking about implementing intelligent features right next to the sensors
and the actuators that form the boundary between the network and the world.
So Chris, what are the benefits to endpoint AI?
The benefits to endpoint AI, well there are a ton of them. Firstly, it's intrinsically more secure,
it's more private. Your data just isn't being sent to the cloud;
it's being processed right at the endpoint itself. And then it saves you a lot
in terms of cost of network bandwidth and for a product designer product user power in turning
on the radio just to transmit stuff. It's amazing how much it costs in energy just to turn on the
radio compared to running your CPU and running your sensors. So your battery operated widget
is going to last a lot longer between charges. That's just a couple of the highlight benefits.
So AI and ML feel like they require a lot of compute to accomplish their tasks, so how's it
possible that some of this can be done on these really tiny power efficient endpoint devices?
That's a great question and I think it's so great, it has two answers. The first is that the
available compute power in microcontroller-scale devices is increasing in leaps and bounds in
recent years and still today. The advent of things like the Cortex-M55 microcontroller
that combines real-time microcontroller behavior with the ability to handle DSP workloads, which is
a very powerful combination. And then there's the Ethos-U55, that's from Arm too and that's
the world's first micro NPU, and it accelerates ML networks and that's really changed the game
as a combination. And the second answer is the software architectures and software techniques
are continually improving. The techniques we can use today to do things like quantizing networks,
optimizing to smaller data types and so on, they mean we can do a lot more with that compute power
than we could only a year or so ago. Both of these things are continuing to improve at an
incredible pace for the foreseeable future. That's where your compute power comes from.
So how do people generally implement AI in terms of silicon IP?
Well at the edge, you've always got this tricky balance, Brian.
You need as much compute power as you can lay your hands on – that's your route to the
smart functionality you really want – but often those devices, they got to be physically small,
their battery powered, they got to be capable of real-time control and real-time actuation.
And the consumer always wants it cheaper, right, so cost constraints figure a lot here.
And that drives the choices that a microcontroller hardware designer is going to have to do. You're
going to have to be running a RTOS and we see leading edge customers opting for platform based
on the Cortex-M55 and they're often using that Ethos-U55 as an accelerator. And the solutions for
higher performance, well they drive you towards application-class processors like Cortex-A55,
and you are going to run something like Linux on those. But the beauty of the current software
architectures we see today is the same ML frameworks scale right across that range.
So a lot of these applications you've talked about weren't possible just a few years ago so obviously
the pace of innovation is accelerating, how do you see endpoint AI evolving in the next few years?
I think product designers are waking up and they are going to continue to wake up
just to the possibilities of what can be done with these new software and hardware
architectures we're talking about and they are coming onto the market right now. And that's
going to transform expectations of what can be achieved. And then secondly, the next generation
because there's always another generation, right. The next generation of platforms that Arm and
other are going to bring to market over the next few years, well that's going to open up another
ton of opportunities for innovative devices and innovative service models. And the pace
of change right now is breathtaking and in my job I see that every single day. But it's fantastic
to be connected to the ecosystem that catalyzes great products ideas and just makes them real.
Well it's nuts for sure, but it's nuts in a really good way. Thanks as always chatting with us,
Chris and in just a few minutes you’ve educated us on a really fascinating topic.
Check out all our other “What Is” episodes here and be sure to subscribe to this channel
because we’ll be adding more as the year progresses. Thanks for listening!