Performing Basic Operations on DataFrames

Welcome back! As you continue your journey in learning PySpark, understanding how to perform basic DataFrame operations is essential. In previous lessons, you learned about creating DataFrames and loading data into them. Today, we will take a step further by exploring some crucial operations: selecting columns, filtering rows, updating existing columns, and adding new columns. Mastering these operations will enable you to manipulate and analyze your data efficiently, making your datasets ready for more complex transformations and analyses.

To begin working with DataFrames, we must set up your PySpark environment by initializing a SparkSession and loading our dataset. In this lesson, we'll use a dataset named "employees.csv", which contains data on employee names, salaries, and departments.

from pyspark.sql import SparkSession

# Initialize a SparkSession
spark = SparkSession.builder.master("local").appName("BasicOperations").getOrCreate()

# Load the dataset
df = spark.read.csv("employees.csv", header=True, inferSchema=True)

# Display the first few rows of the dataset
df.show(3)

Here's a quick look at the dataset:

+-----+------+-----------+
| Name|Salary| Department|
+-----+------+-----------+
|Alice|  3000|         HR|
|  Bob|  3500|    Finance|
|Cathy|  4000|Engineering|
+-----+------+-----------+

With this data, we'll perform key DataFrame operations, including selecting, filtering, updating, and adding columns.

Once your data is loaded into a DataFrame, you may not need every column for your analysis. You can select specific columns using the select method. For example, let's say you're interested in just the "Name" and "Salary" columns from your data.

You can achieve this with the following:

# Select specific columns
selected_df = df.select("Name", "Salary")

# Display the selected columns
selected_df.show(3)

When executed, this code will show you the first few rows of the "Name" and "Salary" columns, helping you isolate the data relevant to your task.

+-----+------+
| Name|Salary|
+-----+------+
|Alice|  3000|
|  Bob|  3500|
|Cathy|  4000|
+-----+------+