Loading Data with Pandas: A Beginner's Guide

Welcome! In this course path, we will explore approaches to handle unbalanced datasets in data science. We will work with a dataset similar to the ones you can encounter in kaggle competitions.

In this course path, we will solve a binary classification task: given information about a customer, we need to predict if they will make a purchase or not.

This path is designed for beginner machine learning engineers who already have basic theoretical knowledge but lack practical experience. To comfortably go through this path, you should be comfortable pandas and some machine learning basics. If you feel you need to learn these first or need a refresh, check these paths:

• Deep Dive into NumPy and Pandas
• Introduction to Machine Learning with SciKit Learn

In data science, data is often stored in formats like CSV, Excel, or databases. To work with this data in Python, we need a way to load it into a format that allows for easy manipulation and analysis. This is where libraries like Pandas come into play. Pandas is a powerful library that provides data structures and functions needed to work with structured data seamlessly. One of its core data structures is the DataFrame, which is essentially a table of data with rows and columns, similar to a spreadsheet or SQL table. Understanding how to load data into a DataFrame is the first step in any data analysis process.

Let's dive into loading data using Pandas. The most common format for datasets is CSV (Comma-Separated Values). On Kaggle, most datasets are presented as .csv files. Pandas provides a convenient function called pd.read_csv() to load CSV files into a DataFrame.

Under the hood, pd.read_csv() reads the file line by line, parses the comma-separated values, and constructs a DataFrame object. By default, it assumes that the first row of the file contains column headers, values are separated by commas, and missing values are represented as empty fields. If your data uses a different delimiter or has no header row, you can specify these options using parameters like delimiter or header.

Here's a basic example:

import pandas as pd

# Load the CSV file into a DataFrame
df = pd.read_csv('data/train.csv')

# Display the first few rows of the DataFrame
print(df.head())

Assuming 'train.csv' contains data, the output will display the first few rows of the DataFrame, giving us a quick glimpse of the data. This function is essential for verifying that the data has been loaded correctly and to get an initial sense of its structure.

The output will be the following:

    age        income  credit_score  ...  membership_status  referral_source  label
0  18.0  53025.508399    708.110039  ...               Gold     Social Media    1.0
1  62.0  47737.465691    633.758381  ...              Basic           Friend    1.0
2  66.0  30737.886739    626.071128  ...              Basic           Friend    1.0
3   NaN  47222.521565           NaN  ...             Silver     Social Media    1.0
4  28.0  30055.350466    703.713381  ...               Gold           Friend    1.0

[5 rows x 13 columns]

Once the data is loaded, it's important to inspect it to understand its structure and contents. The info() method in Pandas provides a concise summary of the DataFrame, including the number of entries, column names, data types, and non-null counts. Here's how you can use it:

# Print information about the DataFrame
df.info()

The output will summarize the DataFrame, showing the number of entries, column names, data types, and non-null counts. This method is particularly useful for identifying any missing values and understanding the data types of each column. Knowing the data types is crucial because it affects how you can manipulate and analyze the data. For instance, numerical operations can only be performed on numeric data types.

When nearly every column has missing values, it's important to check whether certain rows have too many missing fields. Such rows might not be useful for training a model and could be dropped early in the pipeline. This kind of insight often informs your data-cleaning strategy.

Here is the output:

RangeIndex: 900 entries, 0 to 899
Data columns (total 13 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   age                 810 non-null    float64
 1   income              623 non-null    float64
 2   credit_score        681 non-null    float64
 3   num_purchases       881 non-null    float64
 4   time_on_site        826 non-null    float64
 5   num_visits          854 non-null    float64
 6   gender              858 non-null    object 
 7   region              874 non-null    object 
 8   device              891 non-null    object 
 9   preferred_category  763 non-null    object 
 10  membership_status   900 non-null    object 
 11  referral_source     541 non-null    object 
 12  label               900 non-null    int64 
dtypes: float64(7), object(6)

Here is the useful information that we can derive from this:

• Total Entries: 900
• Target Variable: 'label' column is fully populated and can be used for binary classification.
• Features with Missing Data:
  • all columns have missing values.
• Data Types:
  • There are numerical columns (float64)
  • There are categorical columns (object)
• Feature Engineering: Consider imputing missing values and encoding categorical variables for model training.

You'll often want to focus on specific columns within your dataset. In our example, we are working with a dataset that includes a 'label' column indicating the class of each entry; you might want to examine this column more closely. Here's how you can access and print a specific column:

# Print the 'label' column
print("\n'Label' column data:")
print(df['label'])

The output will display the data in the 'label' column. Accessing a column in a DataFrame is straightforward using the column name as a key.

This bracket notation (df['label']) is the most common way to access a column. Alternatively, you could use df.label, but the bracket notation is safer, especially when column names have spaces or special characters, which would cause df.label to break.

This allows you to perform operations or analyses on specific parts of your data, which is often necessary for tasks like feature engineering or data cleaning.

Understanding the distribution of values within a column is essential. The unique() method in Pandas helps identify all unique values in a column. This is particularly useful for categorical data, where you want to know the different categories present. Here's an example:

# Print unique values from the 'label' column
unique_values = df['label'].unique()
print("\nUnique values in the 'label' column:")
print(unique_values)

Output:

Unique values in the 'label' column:
[1. 0.]

The output will list all unique values in the 'label' column. We can see two label types, 0 and 1, which confirms we are dealing with the binary classification task.

In this lesson, we've covered the foundational steps of loading and inspecting data using Python and Pandas. We learned how to load a CSV file into a DataFrame, inspect the DataFrame's structure with info(), access specific columns, and identify unique values within a column. These skills are essential for any data analysis or machine learning project, as they allow you to understand and prepare your data for further analysis.

Now that you've learned the basics of loading and inspecting data, it's time to put your knowledge into practice. In the upcoming practice session, you'll have the opportunity to apply these concepts to explore the given dataset. Let's get started!