Device Memory Allocation

Welcome to CUDA Basics and 1D Operations. To build high-performance GPU applications, we must first master the bridge between two separate worlds: the Host (CPU) and the Device (GPU). Unlike standard C++ where memory is often unified in the programmer's mind, CUDA traditionally requires us to manually manage data across distinct hardware boundaries.

In this lesson, we will establish a professional foundation for GPU programming. You will learn how to use cudaMalloc to carve out space in video memory (VRAM), initialize that memory using cudaMemset, and ensure your code is production-ready using robust error-checking macros.

CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model created by NVIDIA. It allows developers to use the Graphics Processing Unit (GPU) for general-purpose mathematical tasks, a field known as GPGPU (General-Purpose computing on Graphics Processing Units).

While a CPU is designed with a few complex cores optimized for sequential logic and low-latency execution, a GPU is built for massive parallelism. It contains thousands of smaller, specialized cores capable of running the same instruction on different pieces of data simultaneously. To harness this power, we must embrace a specific architecture.

CUDA programs often use a heterogeneous computing model, meaning the application execution is split between two different processors. In the explicit memory management model we are covering in this course:

• The Host: The CPU and its system memory (RAM). It manages the overall control flow and logic.
• The Device: The GPU and its dedicated video memory (VRAM). It handles the intensive data-parallel computations.

In this explicit model, these two spaces are treated as isolated. The Host cannot directly dereference a Device pointer, and the Device cannot directly access Host memory. While CUDA provides advanced features like Unified Memory (which creates a shared virtual address space), mastering the explicit allocation and initialization of data is the fundamental skill required for high-performance GPU engineering.

CUDA programs are written in .cu files, which blend standard C++ with CUDA-specific keywords. To compile them, we use nvcc (NVIDIA CUDA Compiler).

nvcc acts as a "wrapper" compiler:

1. It parses the source file and strips out the GPU-specific code (kernels).
2. It sends the standard host code to a general C++ compiler like gcc, clang, or MSVC.
3. It compiles the GPU code into instructions the hardware understands (PTX or SASS).
4. It links everything into a single executable.

In low-level systems programming, we don't have the luxury of high-level exceptions. Most CUDA functions return a status code of type cudaError_t. If you ignore these codes, your program might continue running with invalid pointers or failed allocations, leading to "silent failures" that are incredibly difficult to debug.

To write professional-grade CUDA, we wrap every call in a macro. This ensures that if a call fails, the program immediately halts and prints a human-readable error message via cudaGetErrorString, along with the exact file and line number.

#include <iostream>
#include <vector>
#include <cstdlib>
#include <cuda_runtime.h>

#define CUDA_CHECK(call)                                                     \
do {                                                                         \
    cudaError_t err = (call);                                                \
    if (err != cudaSuccess) {                                                \
        std::cerr << "CUDA Error: " << cudaGetErrorString(err)               \
                  << " at " << __FILE__ << ":" << __LINE__ << std::endl;     \
        exit(EXIT_FAILURE);                                                  \
    }                                                                        \
} while (0)

With proper error checking in place, we can move on to writing our actual program. First, we'll cover the most important part of any program: memory.

In standard C++, the CPU owns host memory, and you manage it using new or std::malloc. However, the GPU owns device memory, which is a separate address space. Because of this separation, we cannot use std::vector or new for GPU work.

To allocate memory on the GPU, we use cudaMalloc. It functions similarly to std::malloc, but with two key differences:

1. The Return Value: It returns a cudaError_t status code (which we check with our macro).
2. The Pointer: Because the return value is used for the error code, we pass the address of our pointer (float**) so cudaMalloc can write the new GPU memory address into it.

/**
 * Safely allocates memory on the GPU (Device).
 * @param ptr Pointer to the device pointer to be allocated.
 * @param element_count Number of float elements to allocate.
 */
void allocate_device_memory(float** ptr, size_t element_count) {
    // Like std::malloc, cudaMalloc expects the size in BYTES.
    size_t size = element_count * sizeof(float);
    
    // We pass 'ptr' (the address of our float*) so the function 
    // can write the resulting GPU address into our local pointer.
    CUDA_CHECK(cudaMalloc(ptr, size));
}

Once memory is allocated, it contains "garbage" values. To initialize this memory, CUDA provides cudaMemset.

Crucially, cudaMemset is a byte-fill operation, not a typed initialization. It fills a range of memory with a specific 8-bit value. This has important implications for different data types:

• Zeroing Memory: cudaMemset(ptr, 0, bytes) is the most common use case. Since a 32-bit float or a 64-bit double with all-zero bits is interpreted as 0.0, this is a safe and fast way to zero-initialize numerical arrays.
• Non-Zero Values: If you try cudaMemset(ptr, 1, bytes) on a float array, it sets every single byte to 0x01. This results in a 32-bit pattern of 0x01010101, which represents a tiny, garbage subnormal float value, not 1.0f.

Because it works at the byte level, you must always provide the size in total bytes, not the number of elements.

Every cudaMalloc must be paired with a cudaFree. Failing to do so causes a memory leak in VRAM. This is particularly dangerous because GPU memory is often more limited than system RAM, and leaks can persist until the application closes or the driver is reset.

/**
 * Frees memory previously allocated on the GPU.
 * @param ptr Pointer to the device memory to free.
 */
void free_device_memory(float* ptr) {
    if (ptr != nullptr) {
        // cudaFree takes the raw device pointer and marks that memory as available.
        CUDA_CHECK(cudaFree(ptr));
    }
}

Even after allocation and initialization, the CPU cannot "see" what is inside the device pointer. To verify our work, we need a mechanism to copy data back from the GPU to the CPU so we can inspect it.

In the example below, we use cudaMemcpy to act as this bridge. We will explore the mechanics of data transfers and cudaMemcpy much more in-depth in the next unit; for now, consider it a tool to pull our GPU data back into a std::vector for validation.

int main() {
    float* d_data = nullptr; // 'd_' prefix is a common convention for 'Device' pointers
    const size_t element_count = 1024;
    const size_t bytes = element_count * sizeof(float);

    // Host-side vector to hold the data pulled back from the GPU
    std::vector<float> h_check(element_count, -1.0f); 

    std::cout << "Allocating memory for " << element_count << " floats on the GPU..." << std::endl;
    allocate_device_memory(&d_data, element_count);

    // 1. Initialize GPU memory bytes to 0.
    CUDA_CHECK(cudaMemset(d_data, 0, bytes));

    // 2. Copy from Device to Host (Note: we will cover this in the next unit).
    // cudaMemcpy(destination, source, size, direction)
    CUDA_CHECK(cudaMemcpy(h_check.data(), d_data, bytes, cudaMemcpyDeviceToHost));

    // 3. Verify on the Host (CPU).
    bool success = true;
    for (float value : h_check) {
        if (value != 0.0f) {
            success = false;
            break;
        }
    }

    std::cout << "Allocation and zero-initialization: "
              << (success ? "SUCCESS" : "FAILURE") << std::endl;

    std::cout << "Cleaning up and freeing GPU memory..." << std::endl;
    free_device_memory(d_data);

    return success ? 0 : 1;
}

1. Pointer Confusion: Attempting to access d_data[0] directly in the main() function will cause a segmentation fault or crash. The CPU cannot resolve GPU addresses.
2. Elements vs. Bytes: cudaMalloc, cudaMemset, and cudaMemcpy all require sizes in bytes. This is particularly dangerous with cudaMemset; if you pass the element count, you will only initialize the first 1/4th of a float array, leaving the rest as garbage.
3. Silent Failures: Without CUDA_CHECK, a failed cudaMalloc (due to being out of VRAM) would leave d_data as nullptr. Your program would then crash on the next line, making it much harder to pinpoint the original cause.
4. Memset Misconceptions: Remember that cudaMemset fills memory byte-by-byte. It is perfect for zeroing out memory, but it cannot be used to initialize a float array to 1.0f or 3.14f.

We now have a repeatable, professional approach to GPU memory management. By using cudaMalloc to allocate, cudaMemset to initialize, and cudaFree to clean up—all guarded by CUDA_CHECK—we are ready to start processing data. In the upcoming practice, you will implement this pattern to ensure it becomes second nature before we move on to detailed data transfers and writing our first kernels.