Creating New Features from Existing Data

Welcome! In this lesson, we delve into creating derived features, a crucial component for enhancing the value of datasets in data analysis and machine learning. Previously, you've built a solid foundation by learning to encode categorical data and apply mathematical transformations. Now, we focus on creating new features from existing data, adding another layer to your feature engineering toolkit. By learning to derive new features, you'll unlock deeper insights and potentially improve your model's predictive power. Understanding these techniques equips you to better structure data for analysis.

In this lesson, we continue working with the Titanic dataset. We will create new features by focusing on the following columns:

• sibsp: Number of siblings or spouses aboard
• parch: Number of parents or children aboard
• fare: Fare paid for the journey
• age: Age of the passenger

By transforming these features, you'll gain new insights that can enhance survival prediction models.

Let's start with the creation of the family_size feature, representing the total number of family members aboard, including the passenger. This feature is obtained by summing the sibsp and parch columns and adding one to include the passenger themselves. Here's how you can create this feature:

import pandas as pd

# Load the dataset
df = pd.read_csv("titanic.csv")

# Create total family size feature (including the passenger)
df['family_size'] = df['sibsp'] + df['parch'] + 1

# Display the first few rows to verify the new feature
print(df[['sibsp', 'parch', 'family_size']].head())

The output below shows the new family_size column, allowing you to understand passengers' family dynamics onboard:

   sibsp  parch  family_size
0      1      0            2
1      1      0            2
2      0      0            1
3      1      0            2
4      0      0            1

This feature is valuable for survival analysis since families may have had different survival outcomes compared to individuals.

Next, let's create the fare_per_person feature by dividing the fare paid (fare) by the newly-created family_size. This feature provides a per-individual economic perspective:

# Create fare per family member
df['fare_per_person'] = df['fare'] / df['family_size']

# Display the first few rows to verify the new feature
print(df[['fare', 'family_size', 'fare_per_person']].head())

Here's what you observe upon examining the data:

      fare  family_size  fare_per_person
0   7.2500            2          3.62500
1  71.2833            2         35.64165
2   7.9250            1          7.92500
3  53.1000            2         26.55000
4   8.0500            1          8.05000

Analyzing fare per person can provide insights into passengers' socio-economic backgrounds, potentially aiding predictions of survival.

Now that we have created new features from our existing data, it's important to take a closer look at these features to understand their structure and values. By inspecting the initial rows, we can ensure the transformations were applied correctly. Here’s a peek into the first few rows of our modified dataset with the new features included:

# Show new derived features
print("First few rows with new features:")
print(df[['sibsp', 'parch', 'family_size', 'fare_per_person']].head())

This output demonstrates how the new features like family_size and fare_per_person integrate with the existing ones:

First few rows with new features:
   sibsp  parch  family_size  fare_per_person
0      1      0            2          3.62500
1      1      0            2         35.64165
2      0      0            1          7.92500
3      1      0            2         26.55000
4      0      0            1          8.05000

After confirming the new feature creation, let's analyze their statistical properties to grasp the numeric ranges and distributions present. Viewing these summary statistics allows us to understand the nature and variability of the new features. Here we can see key measures such as the average, minimum, maximum, and other statistics that can guide our understanding:

# Display summary statistics of new numerical features
print("Summary statistics of new features:")
print(df[['family_size', 'fare_per_person']].describe())

This statistical overview provides critical insight into the dataset’s new dimensions:

Summary statistics of new features:
       family_size  fare_per_person
count   891.000000       891.000000
mean      1.904602        19.916375
std       1.613459        35.841257
min       1.000000         0.000000
25%       1.000000         7.250000
50%       1.000000         8.300000
75%       2.000000        23.666667
max      11.000000       512.329200

From this statistical summary, we observe that the average family size aboard the Titanic was approximately 1.9, with a maximum family size reaching up to 11 members. Similarly, the fare per person follows a broad range, reflecting significant socio-economic diversity onboard. These insights can be crucial for interpreting different patterns within the dataset and enhancing predictive models.

Congratulations on creating new features! You've added depth to our analysis by crafting the family_size and fare_per_person features from existing dataset elements. These features offer fresh insights and can significantly impact model outcomes by better representing the relationships within the data. As you move to the practice exercises, apply these concepts to solidify your understanding.