Host to Device Transfer

Welcome back to CUDA Basics and 1D Operations. We have already learned how to allocate and free GPU memory safely, so we are now ready for the next key skill: moving real data from Host RAM to Device VRAM, and then bringing results back. In this lesson, we will build small wrapper functions around cudaMemcpy, so transfers become reliable, readable, and easy to verify.

A CUDA program lives in two memory worlds: CPU memory for control and setup; GPU memory for fast parallel work. Since the CPU cannot directly read or write a device pointer, we must explicitly copy bytes across.

A helpful way to think about it is: if the GPU needs $N$ floats, we must send exactly $N \cdot sizeof(float)$ bytes to VRAM, using the correct transfer direction.

As you may recall from the previous lesson, we should not trust CUDA calls without checking their return codes. We will reuse the same CUDA_CHECK pattern so that a failed transfer stops immediately and reports a clear message.

#include <iostream>
#include <vector>
#include <cstdlib>
#include <cuda_runtime.h>

#define CUDA_CHECK(call)                                                     \
do {                                                                         \
    cudaError_t err = (call);                                                \
    if (err != cudaSuccess) {                                                \
        std::cerr << "CUDA Error: " << cudaGetErrorString(err)               \
                  << " at " << __FILE__ << ":" << __LINE__ << std::endl;     \
        exit(EXIT_FAILURE);                                                  \
    }                                                                        \
} while (0)

This gives us two practical benefits: failures are not silent; we also get the exact __FILE__ and __LINE__ number where the problem happened.

We still need device memory to copy into, so we keep small helpers for allocation and cleanup. This is the same idea as Lesson 1, just applied in our transfer-focused program.

void allocate_device_memory(float** ptr, size_t element_count) {
    CUDA_CHECK(cudaMalloc(ptr, element_count * sizeof(float)));
}

void free_device_memory(float* ptr) {
    if (ptr != nullptr) {
        CUDA_CHECK(cudaFree(ptr));
    }
}

Key points to keep straight:

• cudaMalloc needs bytes, not element counts; we multiply by sizeof(float).
• cudaFree is guarded with a nullptr check so cleanup is safe and simple.

Now we implement the main feature of this lesson: a checked wrapper for copying CPU data into GPU memory.

The key element here is cudaMemcpyHostToDevice. This is a constant from the cudaMemcpyKind enumeration defined in <cuda_runtime.h>. It serves as a direction flag, telling the CUDA driver to move data from Host RAM to Device VRAM.

/**
 * Copies data from Host (CPU) memory to Device (GPU) memory.
 */
void transfer_to_device(float* dest, const float* src, size_t element_count) {
    CUDA_CHECK(cudaMemcpy(dest, src,
                          element_count * sizeof(float),
                          cudaMemcpyHostToDevice));
}

This wrapper makes our intent obvious: dest must be a device pointer; src must be a host pointer; the size is computed from element_count.

To validate work or retrieve computed results, we also need the reverse transfer. This function mirrors the previous one, but flips the direction flag to cudaMemcpyDeviceToHost.

/**
 * Copies data from Device (GPU) memory back to Host (CPU) memory.
 */
void transfer_to_host(float* dest, const float* src, size_t element_count) {
    CUDA_CHECK(cudaMemcpy(dest, src,
                          element_count * sizeof(float),
                          cudaMemcpyDeviceToHost));
}

With these two wrappers, our program can move arrays back and forth without repeating low-level cudaMemcpy details everywhere.

Next, we connect everything in main: we prepare host data in a std::vector, allocate device memory, copy to the GPU, copy back, then verify the values match. Since this is a direct bit-for-bit copy without any arithmetic performed, we can safely check for exact equality. Notice how the wrappers keep main focused on the story of the program.

int main() {
    const size_t N = 8;
    std::vector<float> h_input = {1.1f, 2.2f, 3.3f, 4.4f, 5.5f, 6.6f, 7.7f, 8.8f};
    std::vector<float> h_output(N, 0.0f);
    float* d_data = nullptr;

    allocate_device_memory(&d_data, N);

    transfer_to_device(d_data, h_input.data(), N);
    transfer_to_host(h_output.data(), d_data, N);

    bool success = true;
    for (size_t i = 0; i < N; ++i) {
        // Note: Exact equality is safe here only because this is a 
        // bitwise copy; normally, arithmetic requires epsilon checks.
        if (h_output[i] != h_input[i]) {
            success = false;
            break;
        }
    }

    std::cout << "Verified Transfer: " << (success ? "SUCCESS" : "FAILURE") << std::endl;

    std::cout << "Returned Data: ";
    for (float val : h_output) {
        std::cout << val << " ";
    }
    std::cout << std::endl;

    free_device_memory(d_data);
    return success ? 0 : 1;
}

What we should expect when it works is: the verification line reports SUCCESS, and the returned data prints the same sequence we started with.

Verified Transfer: SUCCESS
Returned Data: 1.1 2.2 3.3 4.4 5.5 6.6 7.7 8.8 

1. Swapping Destination and Source: cudaMemcpy follows the standard C memcpy signature: (destination, source, size, direction). Swapping the first two arguments is a common logic error that leads to data corruption or crashes.
2. Wrong Transfer Direction: Using cudaMemcpyHostToDevice when you meant to copy data back to the CPU will cause a runtime error. The direction flag must strictly match the pointer types provided.
3. Elements vs. Bytes: Like cudaMalloc, cudaMemcpy expects the size in bytes. If you pass N instead of N * sizeof(float), you will only copy 1/4th of your data (on systems where a float is 4 bytes).
4. Pointer Confusion: Attempting to access d_data[i] directly within main() (the Host) will cause a segmentation fault. The CPU cannot dereference pointers that belong to the GPU's memory space.
5. Silent Failures: Transfers can fail for many reasons (e.g., passing an invalid pointer or exceeding memory limits). Always wrap cudaMemcpy in CUDA_CHECK to ensure you aren't trying to process "garbage" data that was never actually copied.

We can now reliably prepare GPU-ready datasets by allocating VRAM, copying from host to device, and copying back for validation, all with consistent error checking. This is the exact workflow we will reuse before launching kernels, because kernels assume the data is already in the right place.

Up next, the practice tasks will help us build speed and confidence by writing these transfer steps correctly without needing to look them up.