CUDA tutorial: Difference between revisions

<translate>

[[Category:Software]]

This tutorial introduces the Graphics Processing Unit (GPU) as a massively parallel computing device, the CUDA parallel programming language, and some of the CUDA numerical libraries for use in high performance computing.

{{Prerequisites

* Execute GPU functions (kernels)  

* Transfer data back to the Host memory

Simple CUDA code executed on GPU is called KERNEL. There are several questions we may ask at this point:

* How do you run a Kernel on a bunch of streaming multiprocessors (SMs) ?

3. Copy results from GPU memory back to CPU memory

[[File:Cuda-threads-blocks.png|thumbnail|CUDA block-threading model where threads are organized into blocks while blocks are further organized into grid. ]]

Given very large number of threads (and in order to achieve massive parallelism one has to use all the threads possible) in CUDA kernel, one needs to organize them somehow. in CUDA, all the threads are structured in threading blocks, the blocks are further organized into grids, as shown on FIg. In dividing the threads we make sure that the following is satisfied:

[[File:Cuda_threads.png|thumbnail|Threads within a block intercommunicate via shared memory . ]]

Each thread uses IDs to decide what data to work on:

* Block IDs: 1D or 2D (blockIdx.x, blockIdx.y)

Such model simplifies memory addressing when processing multidimmensional data.

Usually streaming microprocessor (SM) executes one threading block at a time. The code is executed in groups of 32 threads (called Warps). A hardware scheduller is free to assign blocks to any SM at any time. Furthermore, when SM gets the block assigned to it, it does not mean that this particular block will be executed non-stop. In fact, the scheduler can postpone/suspend execution os such block under certain conditions when e.x. data becomes unavailable (indeed, it takes quite some time to read data from the global GPU memory). When it happens, the scheduler takes another threading block which is ready for execution. This is a so called zero-overhead scheduling which makes the execution more stream-lined where SMs are not idling.

There are several type of memories exists for CUDA operations:

* Global memory

* Constant memory

== CUDA Memory Allocation ==

* cudaMalloc((void**)&array, size)

** Deallocates object from the memory. Requires just a pointer to the array.

* cudaMemcpy(array_dest, array_orig, size, direction)

** Copy the data from either device to host or host to device . Requires pointers to the arrays, size and the direction type (cudaMemcpyHostToDevice, cudaMemcpyDeviceToHost, cudaMemcpyDeviceToDevice, etc)

** Same as cudaMemcpy, but transfers the data asynchronously which means it's not blocking execution of other processes.

The following example shows how to add two numbers on the GPU using CUDA. Note that this is just an exercise, it's very simple, so it will not scale up.

<syntaxhighlight lang="cpp" line highlight="1,5">

__global__  void add (int *a, int *b, int *c){

}

int main(void){

int size = sizeof(int);

//  allocate device copies of a,b, c

cudaMalloc ( (void**) &dev_a, size);

cudaMalloc ( (void**) &dev_c, size);

a=2; b=7;

//  copy inputs to device

cudaMemcpy (dev_b, &b, size, cudaMemcpyHostToDevice);

// launch add() kernel on GPU, passing parameters

add <<< 1, 1 >>> (dev_a, dev_b, dev_c);

// copy device result back to host

cudaMemcpy (&c, dev_c, size, cudaMemcpyDeviceToHost);

cudaFree ( dev_a ); cudaFree ( dev_b ); cudaFree ( dev_c );  

}

</syntaxhighlight>

Are we missing anything ?  

That code does not look parallel !

__global__  void add (int *a, int *b, int *c){

</syntaxhighlight>

where blockIdx.x is the unique number identifying a cuda block. This way each cuda block adds a value from a[ ] to b[ ].

[[File:Cuda-blocks-parallel.png|thumbnail|CUDA blocks-based parallelism. ]]

Can we again make some modifications in those triple brackets ?  

<syntaxhighlight lang="cpp" line highlight="1,5">

  Now instead of blocks, the job is distributed across parallel threads. What is the advantage of having parallel threads ? Unlike blocks, threads can communicate between each other ? In other words, we parallelize across multiple threads in the block when massive communication is involved. The chunks of code that can run independently (without much communication) are distributed across parallel blocks.

So far all the memory transfers in the kernel have been done via the regular GPU (global) memory which is relatively slow. Often time we have so many communications between the threads that decreases the performance significantly.  In order to address this issue there exist another type of memory called Shared memory which can be used to speed-up the memory operations between the threads. However the trick is that only the threads within a block can communicate.  In order to demonstrate the usage of such shared memory we consider the dot product example where two vectors are dot-multipled. Below is the kernel:

  <syntaxhighlight lang="cpp" line highlight="1,5">

* Use memcpy times to analyse the execution times

* Always keep CUDA bandwidth in mind when chaning your code

* Know the theoretical peak bandwidth of the various data links

* Utilize the various memory spaces depending on the situation: global, shared, constant

* Constant memory also resides in DRAM- much slower access than shared memory

** BUT, it’s cached !!!