Introduction to Reading Data in Batches with TypeScript

In previous lessons, you learned how to handle datasets stored in compressed formats and manage large datasets efficiently using TypeScript. Building on that foundation, today's lesson will teach you how to read and process data in batches from multiple CSV files using TypeScript. This is important because working with data in smaller chunks, or batches, can make your code more efficient and faster when dealing with large datasets.

Our focus in this lesson will be on a practical scenario where a dataset containing car information is spread across multiple files. You will learn to read, process, and analyze this data to extract meaningful insights, such as determining the car with the lowest price.

In this lesson, we'll work with a set of CSV files containing car data. Here's what a typical record might look like:

• Model: Ford Mustang
• Transmission: Automatic
• Year: 2020
• Price: 25000.00
• Distance Traveled (km): 50000
• Color: Red

These files are divided into multiple parts to allow batch processing, and understanding their structure is crucial as you learn to read and process them efficiently.

Let's delve into reading these CSV files in batches using TypeScript's import syntax and the csv-parser library. We'll build our solution step by step.

First, we need to specify the filenames for our CSV files and prepare a data structure to hold the combined data.

import fs from 'fs';
import csvParser from 'csv-parser';

// Filenames to read
const filenames: string[] = ['data_part1.csv', 'data_part2.csv', 'data_part3.csv'];

// List to store all car data
let carData: { model: string; price: number }[] = [];

Here, we define an array of filenames and create a typed array carData to store all the car data read from the files.

Next, we'll loop through each filename, read the data, and append it to our carData array.

let filesRead: number = 0;
filenames.forEach((filename) => {
    fs.createReadStream(filename)
        .pipe(csvParser())
        .on('data', (row: { model: string; price: string }) => {
            // Convert price from string to float for comparison
            const car = {
                model: row.model,
                price: parseFloat(row.price)
            };
            carData.push(car);
        })
        .on('end', () => {
            filesRead += 1;
            if (filesRead === filenames.length) {
                // Process the combined data here
            }
        })
        .on('error', (error) => {
            console.error('Error reading file:', error);
        });
});

In this snippet:

• We employ fs.createReadStream() to open each CSV file.
• The csv-parser library reads each line, converting it from CSV directly to a TypeScript object.
• We parse row.price from a string to a float for numerical comparison and store the entire row in carData.
• The end event checks if all files have been processed.

After combining the data into the carData array, we can find the car with the lowest price using the reduce method:

if (filesRead === filenames.length) {
    const lowestCostCar = carData.reduce<{ model: string; price: number } | null>((lowest, car) => {
        if (!lowest || car.price < lowest.price) {
            return car;
        }
        return lowest;
    }, null);

    if (lowestCostCar) {
        console.log(`Model: ${lowestCostCar.model}`);
        console.log(`Price: $${lowestCostCar.price.toFixed(2)}`);
    }
}

The reduce method traverses the carData array. It starts with an initial value null, which allows it to hold the first car's data as the lowest initially. For each car, it checks if the current car's price is lower than the stored lowest price. If true, it updates lowest. The result, lowestCostCar, is the car with the lowest price, which we then log. This approach is efficient and clear for accumulating a specific result from an array.

Instead of loading all data into memory, a streaming approach can process each record as it's read. This is beneficial for systems with limited memory or when working with extremely large datasets.

let lowestCostCar: { model: string; price: number } | null = null;
let lowestPrice: number = Infinity;

filenames.forEach((filename) => {
    fs.createReadStream(filename)
        .pipe(csvParser())
        .on('data', (row: { model: string; price: string }) => {
            const price = parseFloat(row.price);
            if (price < lowestPrice) {
                lowestPrice = price;
                lowestCostCar = {
                    model: row.model,
                    price: price
                };
            }
        })
        .on('end', () => {
            filesRead += 1;
            if (filesRead === filenames.length && lowestCostCar) {
                console.log(`Model: ${lowestCostCar.model}`);
                console.log(`Price: $${lowestPrice.toFixed(2)}`);
            }
        })
        .on('error', (error) => {
            console.error('Error reading file:', error);
        });
});

In this implementation:

• We maintain two variables: lowestCostCar to store the data of the car with the lowest price, and lowestPrice, initialized to infinity.
• We parse and compare the price field for each record in the file. If a record's price is lower, we update lowestPrice and store the record in lowestCostCar.
• This approach reduces memory usage since we do not retain unnecessary data.

In this lesson, you learned how to:

• Read data in batches from multiple CSV files using TypeScript's fs module and csv-parser.
• Process that data efficiently and convert data types when necessary.
• Identify specific insights, such as the car with the lowest price, utilizing TypeScript methods like type annotations and robust data handling techniques.

Now, you're ready to apply these skills with practice exercises designed to reinforce your understanding. These exercises will challenge you to read and analyze data from similar datasets efficiently. Continuous practice is key to mastering these data handling techniques.