Reading Data in Batches with Scala

Introduction to Reading Data in Batches with Scala

In previous lessons, you learned how to efficiently manage datasets stored in compressed formats using Scala. Building on that foundation, today's lesson will introduce you to reading and processing data in batches from multiple CSV files using Scala . Processing data in smaller chunks, or batches, is essential for enhancing code efficiency and speed, especially when dealing with large datasets. Our focus in this lesson will be on a practical scenario where car information is spread across multiple CSV files. You'll learn how to read, process, and analyze this data to extract valuable insights, such as identifying the car with the lowest price.

Understanding CSV Data Structure

In this lesson, we will work with a set of CSV files that hold car data. Here is an example of the CSV format: csv model,price,transmission,year,distance_traveled_km,color Chevrolet Silverado,43725.23,Manual,2013,55504,Silver BMW X5,5643.78,Semi-Automatic,2014,11902,Red Honda Accord,42850.79,Manual,2010,102223,Black BMW Series 3,53359.81,Automatic,2009,237231,Gray ... A typical record might look like this: Model : Chevrolet Silverado Price : 43725.23 Transmission : Manual Year : 2013 Distance Traveled (km) : 55504 Color : Silver Understanding the structure of these files is crucial as you learn to read and process them efficiently, especially when dealing with multiple files in batches.

Setting Up for CSV File Batch Reading

To effectively read and process CSV files in batches, we need to set up our environment by defining the necessary classes and data structures using Scala's syntax. We will specify the filenames for our CSV data files. First, we'll define a Car case class to map each row of the CSV file into a Scala object. This class includes fields that correspond to the columns in the CSV file. Scala // Case class to represent a car case class Car(model: String, price: Double, transmission: String, year: Int, distanceTraveled: Int, color: String) Next, we declare an array filenames that will hold the names of the CSV files we want to read. Additionally, we create a list carData to store all the car data read from these files, enabling us to process and analyze this data collectively. Scala // Filenames to read val filenames = Array("data_part1.csv", "data_part2.csv", "data_part3.csv") // List to store all car data var carData = List.empty[Car] By setting up these structures, we prepare to efficiently read and store data from multiple CSV files, allowing us to handle large datasets by processing information in manageable batches.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Reading Data from Each File

Now, we'll loop through each filename, read the file contents, and map CSV records to Car objects using Scala’s capabilities, such as os-lib for file operations and string manipulation to process data.

Scala

// Read and process each CSV file
filenames.foreach { filename =>
  val filePath = os.pwd / filename

  // Read all lines from the CSV file
  val lines = os.read.lines.stream(filePath).toList

  // Skip the header line
  val dataLines = lines.drop(1)

  // Process each line of data
  dataLines.foreach { line =>
    // Split the line into its components
    val Array(model, priceStr, transmission, yearStr, distanceStr, color) = line.split(",")

    // Convert the strings to appropriate types
    val price = priceStr.toDouble
    val year = yearStr.toInt
    val distanceTraveled = distanceStr.toInt

    // Create a Car object with all the attributes
    val car = Car(model, price, transmission, year, distanceTraveled, color)

    // Prepend it to the carData list
    carData ::= car
  }
}

// Read and process each CSV file
filenames.foreach { filename =>
  val filePath = os.pwd / filename

  // Read all lines from the CSV file
  val lines = os.read.lines.stream(filePath).toList

  // Skip the header line
  val dataLines = lines.drop(1)

  // Process each line of data
  dataLines.foreach { line =>
    // Split the line into its components
    val Array(model, priceStr, transmission, yearStr, distanceStr, color) = line.split(",")

    // Convert the strings to appropriate types
    val price = priceStr.toDouble
    val year = yearStr.toInt
    val distanceTraveled = distanceStr.toInt

    // Create a Car object with all the attributes
    val car = Car(model, price, transmission, year, distanceTraveled, color)

    // Prepend it to the carData list
    carData ::= car
  }
}

Here, we read the file and convert each CSV line into a Car object. This is stored in a list for further processing, using Scala’s collections and os-lib for efficient file reading.

With all data combined in carData, the next step is identifying the car with the lowest price using Scala's higher-order functions, like minBy.

Scala

// Check if car data is not empty before finding the lowest cost car
if carData.nonEmpty then
  // Find the car with the lowest price
  val lowestCostCar = carData.minBy(_.price)

  // Output the model and price of the lowest cost car
  println(s"Model: ${lowestCostCar.model}")
  println(f"Price: $$${lowestCostCar.price}%.2f")
else
  // If no valid car data is available, output an appropriate message
  println("No valid car data available.")

// Check if car data is not empty before finding the lowest cost car
if carData.nonEmpty then
  // Find the car with the lowest price
  val lowestCostCar = carData.minBy(_.price)

  // Output the model and price of the lowest cost car
  println(s"Model: ${lowestCostCar.model}")
  println(f"Price: $$${lowestCostCar.price}%.2f")
else
  // If no valid car data is available, output an appropriate message
  println("No valid car data available.")

Using minBy, we can succinctly find the car with the lowest price in carData, demonstrating the power and simplicity of Scala’s functional programming paradigms.

In this lesson, you have learned how to:

Read data in batches from multiple CSV files using Scala and the os-lib.
Process the data efficiently by mapping CSV records to Scala objects using case classes.
Identify insights, such as the car with the lowest price, using Scala's concise syntax and higher-order functions.

These techniques prepare you to handle similar datasets efficiently using Scala. Practice these skills with exercises designed to reinforce your understanding, focusing on efficient data-handling techniques with modern Scala libraries.

Introduction to Reading Data in Batches with Scala

Understanding CSV Data Structure

Setting Up for CSV File Batch Reading

Join the 1M+ learners on CodeSignal

Reading Data from Each File

Finding the Car with the Lowest Price

Summary and Practice Preparation