Hi, everyone, my name is Geoffrey.
And I'm really excited to be here today to talk to you about our work on Skyline.
So this is work that was done at the University of Toronto and the Vector Institute.
And it is joint work with Tovi Grossman and my advisor at the University of Toronto, Gennady Pekhimenko.
So to get started, I'm going to show you a quick teaser video to show you what Skyline can do.
We present Skyline, a new interactive tool for deep neural network training that supports
in editor computational performance profiling, visualization, and debugging.
Skyline's visualizations are linked to specific lines of code.
They update automatically in the background when the code is changed.
And they can also be manipulated, leading to mutations to the batch size in the code
that suggest how to achieve the manipulated performance.
So what is this talk going to be about?
Well, computational performance of deep neural network training has really started to become
an important concern for many deep learning developers.
And as a result, they really need new usable and domain specific performance profiling
tools to help them with this performance debugging.
And so in our work, we make a few observations about the interesting properties of deep neural
network training that can actually help power new interactive features.
The first is that the training is highly repetitive from a computational perspective.
The second is that the performance is actually very predictable.
And by this, I mean the training throughput and memory usage is very predictable with
respect to the batch size.
These are all terms I'm going to describe later in the talk.
And third, that the code users write to implement their models is actually very structured.
And we can leverage all of these properties.
So our idea is to use these properties to bring performance profiling directly into
the editor itself, so that we can work at the same level of abstraction as deep learning users.
So we realized these ideas by building a tool called Skyline.
So Skyline is an interactive in-editor performance profiler that you saw in the video.
So why deep neural networks?
Well, deep neural networks have led to a lot of breakthroughs in many classical machine
learning tasks, such as image classification, machine translation, object detection, and
speech recognition, and many others.
And as a result, deep neural networks have really become widely used throughout industry
and also academia.
One problem with using deep neural networks in practice is that they can be computationally
expensive to train.
So on the left, here, I have a graph from Open AI's AI and Compute blog post.
And they're plotting the amount of compute it takes to train many of the state of the
art neural networks over the years.
So on the x axis are the years and the y axis is the amount of compute it takes in a logarithmic
scale.
So the main takeaway here is that the amount of compute has been increasing exponentially
at a 3.4 month doubling rate.
So this is the amount of compute it takes to train these neural networks.
So one interesting example is open AI, his very own GPT-3 model.
And this has been in the news a few months ago.
So the folks over at Lambda Labs, they actually did an analysis of GPT-3, and they found that
it would take about 355 years to train this model on a commodity graphics processing unit.
And this would amount to about 4.6 million in cloud compute costs.
So clearly, the main takeaway here is that optimizing training performance from a computational
standpoint, is actually still a very important problem.
Unless, of course, you have a lot of resources or a lot of time to spend.
And what's interesting is, if you do a quick search online, you'll actually find numerous
situations of everyday users running into the exact same problem.
They're not sure why their models are training slowly, and they want to figure out what they can do about it.
So before I go any further, I thought it would help to provide some background information
about how deep neural network training actually works.
So an easy way to think about this is to look at it from the perspective of an end-user
who's looking to train a model.
So after they have decided what kind of model they want to train, they have to implement it in software.
So they do this by assembling together a series of layers.
And each of these layers themselves contain parameters that are learnable weights.
So the idea here is that you can adjust these parameters during training to help the model
make better predictions.
In practice, today, almost everyone will use a software framework to do this implementation.
So this includes PyTorch, TensorFlow and MXNet.
And what this means is that they're going to be implementing their models using a high
level language, almost always Python.
After they're done implementing their models, they have to actually go ahead and train them.
And the heavy lifting of training is actually done by high-performance subroutine libraries.
And, of course, the underlying hardware.
Now, what's really interesting here is that the software framework actually provides a
very nice abstraction over these lower levels of the computational stack.
And as a result, deep learning users actually don't need to know too much about the underlying
implementation details.
So they don't need to know too much about the hardware, or the subroutine libraries
to actually train their models in an efficient manner.
So how does training actually work?
Well, conceptually, it's a very repetitive process.
So the idea is to present a series of inputs to the model and have it make predictions.
And then based on the mistakes it makes, adjust the weights in the model so that it can make
better predictions on the next try.
Of course, I'm providing a simplified description here of deep neural network training.
And the idea is to repeat these steps over and over again, until the model achieves a sufficient accuracy.
So what factors actually impact computational performance?
Well, there's actually three main factors.
So the first, of course, is the deep neural network's architecture.
So this is the number of layers that the user uses, the size of the layers, and the type of the layers.
The second are training parameters that the user has to set.
One example is the batch size.
So the batch size is the number of inputs that you present to the neural network during
one step in training.
This is important because prior work had shown that using a larger batch size can lead to
better computational performance.
But this isn't true in every single case.
And third is hardware acceleration.
This is something I alluded to in the previous slide.
So in practice, users tend to use graphics processing units, or GPUs, because they offer
significant performance improvements over CPUs.
But of course, there are many different kinds of GPUs available.
And so depending on which one you use, you may see different computational performance.
In terms of the metrics users look at, they're primarily interested from a computational
standpoint in training throughput.
So this is the amount of data that your model will process per second during training, and
also the memory usage of the model.
This is because your model uses memory during training.
And you need to ensure that you have enough memory on your underlying hardware to actually
be able to support the model in the first place.
If you've run out of memory, you won't actually be able to train the model at all.
So in this work, we focus primarily on the first two factors.
So we assume that the user already has a GPU that they're looking to train on.
And they're mainly interested in understanding what parts of their model might be running
slowly, using up too much memory, and also how they should set the batch size to achieve
better performance.
So when we set out to build Skyline, we wanted to also understand how everyday deep learning
developers dealt with performance today.
So to do this, we conducted a formative study where we interviewed a series of participants
who are graduate student researchers, and practitioners in industry.
So they all had between four and five years of experience working with deep neural networks.
And we asked them open-ended questions about their experiences working with deep neural networks.
I'm not going to be able to get into all the details about our study,
but if you're interested, I'd highly encourage you to check out our paper where you can read more about it.
But I did want to highlight some of the key takeaways from our interviews.
So the first is that the participants actually tended to not use any of the existing profiling
tools, because they found them very complicated and difficult to use.
And as a result, they tended to just guess which parts of their models were running slowly.
Or when they really needed to measure the performance of a particular part of their
model, they would manually instrument their code.
The graduate student researchers pointed out to us that performance was very important
to them, because they tended to be constrained by the hardware resources that they had available,
constrained in terms of what kind of models that they could experiment with.
So they really wanted to get the most out of the hardware that they had.
And third, the participants let us know that they were interested in getting more visibility
about the memory usage of their models.
They thought that this would help them shape their layers and also choose batch sizes to use.
So, why did users have a lot of trouble working with existing profiling tools?
If you recall from the previous slides, users really work at a high level of abstraction.
So they implement their models in Python, with the help of software frameworks.
But as we know, performance really depends on the entire software and hardware stack.
And as a result, many of the existing tools work on a low level in the stack.
And this means to use them effectively, users really need to have a good understanding of
the lower levels of the stack.
And one of the consequences of this is that it makes it very difficult to correlate the
high level code they write with the performance data they see in these tools.
For example, on the left, here is Python code for PyTorch, and on the right is NVIDIA's visual profiler.
And it's very difficult for users, for example, to know how this line of Python code they
might write connects to the performance traces that they see in the visual profiler.
So in our work, we make the argument that users really should have high level profiling
tools too.
When you look at the diagram here, you'll notice that there's actually space at the
top of the stack for a profiling tool that works at the same level of abstraction as
deep learning users.
And so we think we can do a lot better if we take advantage of what actually makes deep
neural network training special.
So we make a series of observations about the useful properties of deep neural network training.
The first is that the computation is actually highly repetitive.
And what this means is that we can profile the computational performance of a training
process very quickly, because we only need to look at a few of the training steps.
The second is that performance and by this I mean throughput and memory usage is very
predictable with respect to the batch size.
And the third is that the code users write is actually very structured.
And we can take advantage of these properties to support new interactive features---profiling features
that can live inside the editor.
And so that's where Skyline comes in.
So Skyline is an interactive in editor computational performance profiler for deep neural network training.
So Skyline works with models written in PyTorch, and it is a plugin for Atom.
And as you can tell from the video, one of the key ideas behind Skyline is to bring domain
specific profiling information directly into the editor itself, where users do their development.
So I'm going to spend the next few slides going over some of Skyline's features.
So Skyline's visualizations are interactive.
So when hovering over any of the visualizations, Skyline will actually highlight the relevant
lines of code that are associated with that visualization.
And this helps users connect their performance data to the code that they actually wrote.
This is made possible by the structured nature of the code, because users have to write their
models and implement them in a certain way.
This makes it a lot easier for Skyline to connect the performance data with the code
that users write.
The second is that Skyline's visualizations are manipulatable.
So when you drag them Skyline will actually suggest batch sizes that you should use to
potentially achieve that manipulated performance.
This is made possible by empirically derived models for the memory usage and throughput
versus the batch size.
And this helps users select batch sizes in a much more informed way because they can
actually visualize the potential performance impact of different batch sizes they might use.
And third Skyline really lends itself to this notion of live and proactive performance debugging.
So when users change their code in the editor and save it Skyline will actually re-profile
the model in the background to present updated performance information.
So this will help users explore performance during development, so that it's no longer
something that is an afterthought, rather something that you can play around with while
you are developing your models.
So to evaluate Skyline, we conducted an exploratory qualitative user study.
So we recruited seven graduate students in deep learning, and we had them complete a
series of timed diagnostic tasks using Skyline.
So this included things like figuring out which layers took the longest to run, and
which layers used up the most memory, for example.
And after they did this, we had them fill out a survey about the usefulness and ease
of use of each of Skyline's features.
So again, I'm not going to be able to get into all the details of our study.
So if you're interested in reading more, I'd encourage you to check out our paper.
Overall, we received a lot of very positive feedback.
And actually three out of the seven participants voluntarily mentioned to us that they wanted
to use Skyline going forward in their work.
So there's plenty of other details that I didn't get a chance to cover in this talk.
So this includes more details about our interviews, details about our qualitative user study,
information about how we implemented Skyline and how the performance prediction models work.
And also some discussion about related work, existing tools, and future directions for Skyline.
So I wanted to leave you with some of the key takeaways that I hope you got from this talk.
So first is that computational performance is really an important concern for deep neural network training.
The second is that Skyline leverages properties of deep neural network training to really
power its interactive features.
And looking a bit more broadly, if you leverage domain specific characteristics of a workload,
you might be able to power new features that perhaps before were either infeasible or just not so practical.
And so that's Skyline. I wanted to also mention that Skyline is open source software.
So if you're interested in contributing or learning more, you can head to our website
which is shown at the bottom of the slide.
You can also install Skyline today through pip and through the Atom package manager.