Writing Data in Batches with Rust

Writing large data in smaller chunks, or batches, can be a powerful strategy in Rust for boosting performance and managing memory usage more effectively. By processing data in pieces, you can prevent overwhelming your system’s resources. This is especially useful when you’re dealing with content that’s too big to handle all at once. Rust’s strong guarantees around safety and performance make it well-suited for these scenarios, helping you avoid unnecessary complexity while efficiently writing data to files.

In this lesson, you’ll learn how to:

• Generate a sizable dataset of random numbers.
• Write these numbers to a file in batches.
• Verify that the file contains the expected number of lines.

Let’s dive in! ⚙️

Before creating and writing data in batches, you need to define a few parameters to control how many batches you want to create, how many rows each batch should contain, and how many columns each row should have. You’ll also set up your random number generator.

Below is a snippet demonstrating how to set up those configurations and create the skeleton of our program:

use std::fs::{File, OpenOptions, create_dir_all};
use std::io::{BufWriter, BufRead, BufReader, Write};
use std::path::Path;
use rand::Rng;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Configuration constants
    const NUM_BATCHES: usize = 5;
    const BATCH_SIZE: usize = 200;
    const NUM_COLUMNS: usize = 10;

    // Your CSV file name (replace with any name you want)
    let file_path = Path::new("output.csv");

    // Create the directory if it doesn't exist
    if let Some(parent) = file_path.parent() {
        create_dir_all(parent)?;
    }

    // Open the file for writing/appending
    let file = OpenOptions::new()
        .create(true)
        .write(true)
        // Set truncate(true) if you wish to overwrite existing content
        .truncate(true)
        .open(file_path)?;

    // Wrapped in a BufWriter for efficient writes
    let mut writer = BufWriter::new(file);

    // Initialize a random number generator
    let mut rng = rand::thread_rng();

In the snippet above, you define how many total batches you’d like to produce, how many rows each batch should contain, how many columns each row will have, and where you’d like to write your data. You also prepare a BufWriter for efficient writes and a random number generator for producing random numeric values. BufWriter reduces the number of actual write operations to the disk by buffering data in memory first. This is especially useful when writing many small chunks (like rows in a CSV), as each write! call would otherwise trigger a separate and potentially costly system call. By using BufWriter, multiple small writes are batched into larger, fewer I/O operations, improving overall performance.

With your preparation in place, now you can generate the random data and write it in a batched manner. Each batch ensures you're not handling all data at once, which helps keep memory usage manageable and makes your process more efficient.

Here’s how to generate and write your data:

    // Write data in batches
    for batch in 0..NUM_BATCHES {
        for _ in 0..BATCH_SIZE {
            // Write one row
            for col_index in 0..NUM_COLUMNS {
                // Generate a random floating-point value
                write!(writer, "{:.2}", rng.gen::<f64>() * 100.0)?;

                // Add comma unless it's the last column
                if col_index < NUM_COLUMNS - 1 {
                    write!(writer, ",")?;
                }
            }
            writeln!(writer)?;
        }

        // You can flush after every batch to ensure data integrity
        writer.flush()?;

        println!("Batch {} written.", batch + 1);
    }

Inside the loop:

• You iterate over the total number of batches.
• For each batch, you generate multiple rows of data (each containing the specified number of columns).
• The write! and writeln! macros efficiently handle line-based output, and you flush each batch to ensure the data is safely written to disk.
• flush() forces any buffered data to be written to disk immediately. While Rust automatically flushes on drop, doing it manually after each batch helps ensure that if the program crashes mid-execution, data up to the last batch is preserved. This is particularly important in long-running or resource-intensive processes.

After writing the file, it’s good practice to verify that you’ve successfully appended the correct number of lines. By double-checking this count, you confirm that your batch process worked as expected.

Below is an example of how to verify the final row count:

    // Open the file again in read-only mode
    let file = File::open(file_path)?;
    let reader = BufReader::new(file);

    // Count the lines
    let line_count = reader.lines().count();

    println!("The file has {} lines.", line_count);

    Ok(())
}

In this snippet, you:

• Re-open the file for reading.
• Wrap it in a BufReader to read lines easily.
• Use the count method to iterate through all lines, verifying the expected number of rows.

In this lesson, we explored how to efficiently write large volumes of data in Rust by using batching techniques. By chunking up the data, you can mitigate potential performance bottlenecks and decrease memory usage. We covered everything from configuring batch sizes, creating data with random values, writing it in a structured format, and verifying that the output file contains as many lines as intended.

Rust’s robust standard library and crates like rand make it easier to handle such tasks in a safe, performant manner. Try experimenting with different batch sizes and column counts to see how it impacts performance. With this approach, you can confidently tackle larger datasets in your Rust applications without running into scalability issues.

Keep practicing and explore reading your data back in next, to gain even more control over your data pipelines. Happy coding!