Hi and welcome to Arm’s “What Is” program. In each episode, we dive into a tech topic
to give you insight and perspective into some of the hottest design trends today.
I’m Brian Fuller, editor-in-chief at Arm and today we’re going to explore
AI Inference or to spell it out, artificial intelligence inference.
AI is obviously a hot topic and a growing application space. But
it can seem like a complex technology landscape. Today to help us understand one aspect of that
landscape – AI Inference are two of my favorite people in this space, Dennis Laudick and Steve
Roddy. Dennis is head of AI and machine learning marketing at Arm, and Steve is head of products
for AI and ML. And here’s something a lot of people don’t know about these guys:
Dennis got an engineering degree but he minored in philosophy. That comes in handy in the AI
world. Now Steve dabbles in a lot of different sports and for his efforts, he’s broken 13 bones.
Let's dive right in. I'm going to start with Steve.
You're looking good today my friend. What is AI Inference? And how does it work?
So in the world of AI, or more specifically machine learning, there are two phases if
you will. There's the training process. That's the process of actually developing
particular piece of AI. A and then when you deploy it and use it in a real-life situation on new data
that's called inference. So training based on recorded, stored, and labeled images or sound
or data and then deploy in a new situation and use the AI to help augment human decision.
So at the technology level, how is inference
implemented? Is this done on CPUs or specialty processors or something else?
Yeah, that's an interesting one. I think that the industry again is a little bit confused about this
certainly they have been and some clarity coming out of it. Machine learning can run on any type of
processor, it just depends on what you are trying to do. So machine learning is just computing,
it's the same thing we've been doing for decades now. The difference with machine learning – or
what is often called artificial intelligence today – is there's a massive upshoot of particular types
of computing they want to do like many, many words of magnitude. And so if you want to do that really
complex version or high performance or really high efficiency that's when you can get benefits out
of using a computing architecture that's more catered to those type of workloads,
which is what you get with Arm's CPUs or GPUs and our NPUs. They are all kind of
different variants depending on what you are trying to achieve with your machine learning.
Got it. Got it. And Steve, is AI inference done
mostly in the cloud or is it starting to move out to the edge and endpoints?
Good question. The easiest way to think of it is the best place to run your model to do your
inference is where your data is located. So certain problems you got all the data
stored in the cloud and that could be something like some financial analysis analysts on Wall
Street looking at stock trading patterns the last 20 years and trying to predict where the stock
goes tomorrow morning. Well obviously all the data is stored on the servers or all the ways that
machine learning is used to make recommendations when you log into your favorite shopping site.
Your data is naturally in the cloud that's where you want to run it. There's plenty of
other applications where the data is being sent sensed somewhere out in the real world, out on
the edge or the endpoint as they say. So if you have aa smart doorbell or smart door lock that
recognizes your face as you walk up to it and wants to unlock your door you want to do it there.
You do it there because it's more reliable there and you don't have to rely on internet connection,
you don't have to beam the data back to a data center somewhere or worry about your privacy,
worry about the latency of it coming back. So the rules evolve to apply the inference at the point
and time where data is being created or stored, that makes it more energy efficient rather than
try to beam streams of video back and forth from cameras to cloud, etc. So that's the set up.
So obviously the relentless march to technology is enabling us to deploy this technology farther
and farther out. Dennis, how do you see you know given that scale and advancement the last
few years, how do you see AI inferencing evolving in the next three to four years?
Yeah, I mean I'm a little biased because I work in the AI industry, but I really do believe that
over the last five years or so I think Steve and I have seen an explosion of application out on
the devices and that's obviously what we spend a lot of our time focused on is enabling that, but
we're nowhere near seeing where this technology can go. So
recently we've seen the early phrases were about performance and accuracy. More recently
what we've seen about machine learning is a focus on efficiency and making things
smaller and being able to put them on devices around you – even tiny sensors get better with
AI. We see this stuff going everywhere. It's really exploding across the Arm device world.
Well thanks gentlemen! I think we think know a lot more about AI inference than we did just a
few minutes ago. Check out all our other “What Is” episodes here and be sure to subscribe to
this channel because we’ll be adding more as the year progresses. Thanks for watching!