A graphics processing unit (GPU) contains an array of compute units called streaming multiprocessors (SMs). Each SM controls multiple cores, each capable of handling multiple simultaneous threads.
The GPU memory consists of registers private to threads, L1 cache and shared memory used across threads within each SM, and L2 cache and global memory shared by all SMs.
The GPU's job is to run as many workloads as possible in parallel via this hierarchical organization. The code running on a GPU core is called a kernel, which is often written in CUDA. These kernels are scheduled by:
- Grouping threads into warps, which are synchronized to run threads simultaneously on different parts of data.
- Grouping warps into blocks, which each block assigned to a single SM. That SM may have other blocks as well and can run multiple in parallel—though some may wait while others run.