Most CUDA developers are aware of the cudaMalloc and cudaFree API capabilities to allocate GPU accessible memory. Nevertheless, there has long been an impediment with these API functions: they aren’t stream ordered. On this put up, we introduce new API features, cudaMallocAsync and cudaFreeAsync, that allow memory allocation and deallocation to be stream-ordered operations. Partially 2 of this sequence, we spotlight the benefits of this new functionality by sharing some huge information benchmark results and supply a code migration guide for modifying your present applications. We additionally cover advanced subjects to reap the benefits of stream-ordered memory allocation within the context of multi-GPU access and the usage of IPC. This all helps you improve performance inside your current purposes. The next code example on the left is inefficient as a result of the primary cudaFree name has to wait for kernelA to complete, so it synchronizes the machine earlier than freeing the memory. To make this run more effectively, the memory could be allotted upfront and sized to the larger of the 2 sizes, as shown on the proper.

This will increase code complexity in the appliance as a result of the memory management code is separated out from the enterprise logic. The problem is exacerbated when different libraries are concerned. This is much more durable for the appliance to make environment friendly as a result of it may not have full visibility or management over what the library is doing. To circumvent this downside, the library would have to allocate Memory Wave when that perform is invoked for the first time and never free it till the library is deinitialized. This not only will increase code complexity, but it surely also causes the library to carry on to the memory longer than it must, probably denying another portion of the application from utilizing that memory. Some applications take the concept of allocating memory upfront even further by implementing their very own customized allocator. This adds a big quantity of complexity to software growth. CUDA goals to supply a low-effort, high-efficiency various.

netzero.net

CUDA 11.2 introduced a stream-ordered memory allocator to unravel these kind of problems, with the addition of cudaMallocAsync and cudaFreeAsync. These new API capabilities shift memory allocation from international-scope operations that synchronize the whole system to stream-ordered operations that enable you to compose memory management with GPU work submission. This eliminates the need for synchronizing outstanding GPU work and helps limit the lifetime of the allocation to the GPU work that accesses it. It is now potential to handle Memory Wave Program at operate scope, as in the following instance of a library operate launching kernelA. All the usual stream-ordering guidelines apply to cudaMallocAsync and cudaFreeAsync. The memory returned from cudaMallocAsync may be accessed by any kernel or memcpy operation as long as the kernel or memcpy is ordered to execute after the allocation operation and before the deallocation operation, in stream order. Deallocation will be performed in any stream, as long as it is ordered to execute after the allocation operation and after all accesses on all streams of that memory on the GPU.

In effect, stream-ordered allocation behaves as if allocation and free have been kernels. If kernelA produces a valid buffer on a stream and kernelB invalidates it on the same stream, then an application is free to access the buffer after kernelA and before kernelB in the appropriate stream order. The following instance exhibits various valid usages. Determine 1 reveals the assorted dependencies specified in the sooner code example. As you'll be able to see, all kernels are ordered to execute after the allocation operation and complete before the deallocation operation. Memory allocation and deallocation can't fail asynchronously. Memory errors that happen due to a call to cudaMallocAsync or cudaFreeAsync (for example, out of memory) are reported immediately by an error code returned from the call. If cudaMallocAsync completes efficiently, the returned pointer is guaranteed to be a legitimate pointer to memory that's safe to entry in the suitable stream order. The CUDA driver makes use of memory pools to realize the conduct of returning a pointer immediately.

Edit

Pub: 02 Sep 2025 14:14 UTC

Views: 4