Welcome back Memory footprint of AI models, or data
in general, is exploding , and if we can solve the compute demand by
adding more cores or more compute nodes, This is not easy to be done when expanding
the memory. As memory has to be shared sometimes between
compute nodes and data-structures are not easy to be broken
++One of the capabilities that SYCL add to C is the handling of disjoint memories
Memory spaces of a GPU, FPGA or CPU looks totally different, mapped differently
But for me as a developer. I don’t care, I want it all to be abstracted for me..
And SYCL can do it, let me show you how.
SYCL allows you to handle memory accesses using buffers or a unified shared memory or
images I’ll talk about images in a future video.
In previous video we connected all the dots into a full working SYCL program
We used buffers to explicitly move data between host and device memories,
I’ll show you the example and how to use unified shared memory instead to let the SYCL
runtime implicitly handle some of this data movement.
I’ll also leave you the code snippets, look for the link in the video description.
So we have a SYCL queue, attached to our GPU selector
And now I can submit work to this Q.
The code inside the curly brackets can run on any device I choose.
Here on my GPU as my Q is attached to the GPU
And now to the data.. I allocate here a buffer of integers, R is
the range..
I am using an accessor to access that buffer
And this array, A, is visible both to the host. To the CPU
And to the accelerator, the GPU in this case
The actual, physical data transfers between the host memory and the device memory, GPU
in my case Are managed by the run-time.
So when you define the buffer in the host, it doesn’t mean that the device already
see that data But he will have it exactly when he needs
it..
I find the usage of buffers to be very elegant, but you can also use USM..
Unified shared memory
So instead of allocating a buffer, I’m using here the good old MALLOC
In this case I am using the SYCL malloc_shared function to inform the runtime that allocated memory
is shared between the host and the accelerator
An interesting observation is that memory allocation using USM is tied to a particular
SYCL Q.. And therefore to a particular device..
Buffers are more flexible as they are not tied to any single device..
They can be accessed anywhere
And now I can use the data in the GPU attached to that Q.
So basically, we have here one unified shared memory between the host and the device mapped
to Q At least from the programmer point of view.
Pretty readable and close to your current C++ code style
BTW, there are other syntaxes that can work here, pls check the manual.
And you can see that I do not need to use "submit", I can use a more condensed syntax..
Malloc_shared is probably the cleanest way to use unified shared memory,
It does not mean that it is the fastest… you basically leave the memory coherency and
management to the run-time
If you prefer more control you can use malloc_host or malloc_device
and by that, only the host or the device will own the data, which reside in host or device
memory.
And you will need to transfer the data between the devices,
you’ll have to do it explicitly using mem-copy
And here is a very simple, clean example, Again, find the link to the code below.
Data is an array on the host, initialized
Device-data exist only on the device, as we use here malloc device..
I’ll mem-copy the data into the device data So I’m copying the data from the CPU to
the GPU
Pay attention that SYCL queues are asynchronous and non-blocking
So for memory coherency, you need to wait for all the data to be transferred..
Before you start manipulating it..
Inside the kernel, on the GPU, I work on device-data.
And after I finish, I have to explicitly copy it back to the CPU data
Mem-copy again.
Another difference from buffers.. Please pay attention the SYCL buffers have
destructors but USM has to be freed like any other dynamically
allocated memory. So just add a free command to release the
memory
So memory handling using buffers, or unified shared memory,
Try it out, thank you.