Reading and Processing CSV Data in Batches with Rust

When dealing with multiple CSV files, an efficient way to handle large amounts of data is to process it in manageable chunks or batches. In this lesson, you’ll learn how to read and merge information from several CSV files, all while keeping your memory usage in check. You’ll also practice finding the lowest-priced item (in this case, a car) from the combined dataset. This approach demonstrates how Rust’s standard library and crates can streamline the task of ingesting and processing large data with minimal overhead.

For this lesson, each CSV file contains information about cars using columns such as model and price. Here’s a simple snippet of what a CSV row might look like:

model,price  
Toyota Corolla,19500.00  

In Rust, we’ll represent this data with a struct to store each row’s information. By focusing on just the fields you need — model and price — you can simplify your parsing logic and keep your code lightweight.

Below is an example of the struct used to capture each row in memory:

#[derive(Debug)]
struct Car {
    model: String,
    price: f64,
}

To gather data from multiple files, you can list those files in a small array, then iterate through each entry. You’ll also need a data structure (like a vector) to keep track of all the cars you read across these files.

Below is a snippet that sets up the list of filenames and initializes a mutable vector to store your data:

let filenames = [
    // Your CSV file names go here
];

let mut car_data = Vec::new();

Once you’ve organized the file names, you can parse each file using the csv crate, which provides a convenient Reader for handling CSV data. This crate automatically handles splitting rows by columns and can iterate over the resulting records.

In Rust, reading data from each file and converting it to the Car struct is straightforward. You’ll open each file in turn, create a CSV Reader, then go through the records. Whenever you successfully parse the relevant columns, you push the resulting struct into your data vector.

Below is a snippet showing how you might accomplish this:

use std::fs::File;
use std::path::Path;
use csv::Reader;

for filename in &filenames {
    let file_path = Path::new(filename);
    match File::open(file_path) {
        Ok(file) => {
            let mut reader = Reader::from_reader(file);
            
            // Process each record
            for result in reader.records() {
                match result {
                    Ok(record) => {
                        // Assuming the data has at least two columns: model and price
                        if record.len() >= 2 {
                            if let Ok(price) = record[1].trim().parse::<f64>() {
                                car_data.push(Car {
                                    model: record[0].trim().to_string(),
                                    price,
                                });
                            }
                        }
                    },
                    Err(e) => println!("Error reading record: {}", e),
                }
            }
        },
        Err(err) => {
            println!("Error opening file: {}", err);
            continue;
        }
    }
}

If you need to generate some CSV files for demonstration or testing, you can create a small function that writes out CSV-format text. This approach keeps your main logic clean while enabling you to quickly spin up sample data without manual preparation:

use std::error::Error;

fn create_sample_data_files() -> Result<(), Box<dyn Error>> {
    // Example data, each row has a model and price
    let part1_data = "Model, Price\n\
                      Toyota Corolla, 19500.00\n\
                      Honda Civic, 21000.50\n\
                      Ford Focus, 18750.75";
    
    let part2_data = "Model, Price\n\
                      BMW 3 Series, 42000.00\n\
                      Mercedes C-Class, 43500.25\n\
                      Audi A4, 39999.99";
    
    let part3_data = "Model, Price\n\
                      Hyundai Elantra, 17250.50\n\
                      Kia Forte, 16800.00\n\
                      Mazda 3, 19200.75";
    
    // Write data to files
    // (In a real project, choose your file names or directories)
    std::fs::write("data_part1.csv", part1_data)?;
    std::fs::write("data_part2.csv", part2_data)?;
    std::fs::write("data_part3.csv", part3_data)?;
    
    Ok(())
}

Once your data is loaded into a vector of Car structs, you can easily locate the car with the lowest price using Rust’s iterator methods. By calling iterator functions like min_by and wrapping partial comparisons in a closure, you can seamlessly filter for the minimum element:

use std::cmp::Ordering;

if let Some(lowest_cost_car) = car_data.iter().min_by(|a, b| {
    a.price.partial_cmp(&b.price).unwrap_or(Ordering::Equal)
}) {
    println!("Model: {}", lowest_cost_car.model);
    println!("Price: ${:.2}", lowest_cost_car.price);
} else {
    println!("No valid car data available.");
}

In this lesson, you learned how to:

• Set up a struct in Rust to represent each row of a CSV file.
• Batch-process data by listing multiple files and iterating through them with the csv crate.
• Parse and load records into a vector of structs.
• Use iterator methods such as min_by to identify the item with the lowest value.

With these techniques, you can comfortably handle data across multiple CSV files, extracting the information you need for further processing. Now is the perfect time to practice by experimenting with different datasets, adding additional fields to your struct, or applying filters and aggregations. By doing so, you’ll reinforce the fundamental Rust patterns for data ingestion and batch processing. Have fun, and enjoy your journey into efficient file handling in Rust!