Transforming and Saving Data with RDDs | CodeSignal Learn

Transforming and Saving Data with RDDs | CodeSignal Learn

Introduction to Transforming and Saving Data with RDDs

Welcome to the final lesson of this first course on PySpark and RDDs . In this concluding lesson, we will focus on a practical task: transforming sales data to identify high-value transactions and saving the results to a file. This task will help you grasp the concept of real-world data processing. By the end of this lesson, you'll know how to filter unwanted data, perform transformations to extract necessary information, and save the results for future use. Let's dive in!

Setting Up SparkSession and Loading Data into an RDD

Before we begin transforming data, we ensure our PySpark environment is ready by initializing a SparkSession and reading the sales data from a file named sales.txt into an RDD. Here's how we accomplish both tasks: Pythonfrom pyspark.sql import SparkSession # Initialize SparkSession spark = SparkSession.builder \ .master("local") \ .appName("SalesDataProcessing") \ .getOrCreate() # Create an RDD by reading from a sales data file sales_rdd = spark.sparkContext.textFile("sales.txt") # Show the first 5 items of the RDD print(sales_rdd.take(5)) This combined code sets up our Spark environment, reads each line of sales.txt into an RDD, and then prints the first 5 items for inspection. If we have a file containing different items with prices in each line, here is what the print output might look like: text['Laptop,900', 'Mouse,20', 'Monitor,150', 'Keyboard,30', 'Smartphone,700'] With the data now loaded into an RDD, we’re ready to begin transforming it to identify and extract high-value sales transactions.

Filtering High-Value Sales

Filtering is essential to isolate data entries of interest. In this step, we focus on high-value sales, defined here as transactions over $100. Utilizing the filter transformation allows us to apply a lambda function to each RDD element and retain only those meeting the condition. Python # Use filter to retain only sales entries with amounts greater than $100 high_value_sales_rdd = sales_rdd.filter(lambda line: float(line.split(",")[1]) > 100) In this code, the filter() function works by parsing each line, splitting it by the comma delimiter, and checking if the second part (the amount) exceeds 100. This operation transforms sales_rdd into high_value_sales_rdd, which now contains only the desired high-value transactions.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Extracting Product Names

After identifying high-value sales, the next step involves transforming this data to extract product names. This is where the map transformation comes in handy, allowing us to apply a function to each element of the RDD to transform its data. # Use map to extract product names from high value sales, assuming format: "Product,Amount" product_names_rdd = high_value_sales_rdd.map(lambda line: line.split(",")[0]) The map() function here splits each line again, but instead of checking values, it extracts the first part, the product name. product_names_rdd now holds these product names, extracted from the filtered high-value sales.

Saving the RDD Data

The final step is to save the transformed data. Instead of saving to a single text file, PySpark's saveAsTextFile writes data to a directory. This is because PySpark operates in a distributed manner, often resulting in multiple output partitions, each saved as a separate file within the directory. # Save the extracted product names to a new text directory product_names_rdd.saveAsTextFile("output_products") In this code, saveAsTextFile creates a directory named output_products . Within this directory, the data is partitioned into one or more files, typically named part-00000, part-00001, etc., which correspond to the number of partitions in the RDD. Here is an example of what the directory structure might look like: output_products/ ├── _SUCCESS ├── part-00000 ├── part-00001 └── ..._SUCCESS is a marker file indicating that the save operation was successful. part-00000, part-00001, etc., contain the actual data, each representing a partition of the RDD.

Reading and Verifying the Saved Data

To ensure the data has been correctly saved and to preview the results, we can use the textFile method to read all part files at once. # Read the content of all parts in the saved directory saved_rdd = spark.sparkContext.textFile("output_products") # Collect and print the first three products print(saved_rdd.take(3)) This approach ensures that we load the entire dataset stored across all part files, providing a complete view of the saved results. When running this code, you might see an output similar to the following: ['Laptop', 'Monitor', 'Smartphone'] This output confirms that the high-value product names— Laptop, Monitor, and Smartphone —have been successfully extracted and saved.