Welcome back! In the previous lessons, we explored foundational techniques in feature engineering, focusing on encoding categorical data and converting continuous data into discrete categories through feature binning. Now, we will delve into another vital technique: applying mathematical transformations to data. These transformations are essential for modifying data distribution, handling skewness, and ultimately improving the performance of machine learning models. Today, you'll learn how to apply log, square root, and cube root transformations specifically to the 'fare' column in the Titanic dataset. This lesson is part of our broader objective to shape and transform features effectively, building directly on your prior knowledge.
Why Apply Mathematical Transformations?
Mathematical transformations are a powerful tool in data preprocessing. They help to stabilize variance, normalize distributions, and make patterns more visible, which can enhance model performance. For instance, log transformation is commonly used when data exhibits exponential growth or right skewness, as it compresses the range of variable values, pulling high values closer and magnifying low values. Square root and cube root transformations are useful in reducing skewness of a moderate nature. By applying these transformations, you can make your data more suitable for modeling and ensure that features are on an appropriate scale. Understanding when and why to apply these transformations is key to effective data preparation.
Loading and Exploring the Dataset
Log Transformation
Square Root Transformation
Cube Root Transformation
Comparing Transformations
Upon applying these transformations to the 'fare' column, let’s compare their impact on variance to evaluate their effectiveness:
Each transformation offers a unique mechanism to refine data distribution based on specific modeling goals.
Review and Summary
Congratulations on advancing to this stage of the course! In this lesson, we covered the theory and practice of applying mathematical transformations like log, square root, and cube root to manipulate data distributions. By working through the 'fare' column of the Titanic dataset, you learned how these transformations adjust data for better pattern visibility and model performance. These skills are essential in your data preprocessing toolkit.
As you progress to the practice exercises, you'll have the opportunity to apply these transformations to datasets yourself, solidifying your understanding by transforming data features independently. This hands-on practice will not only reinforce today's concepts but also prepare you for more advanced feature engineering tasks in future studies. Keep up the excellent work, and enjoy the practice!
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Let’s start by loading and exploring the dataset, focusing specifically on the 'fare' column from the Titanic dataset to set the stage for our transformations. This initial exploration helps us understand the data distribution and determine the need for transformations.
import pandas as pdimport numpy as np# Load the datasetdf = pd.read_csv("titanic.csv")# Original fareprint("Original Fare:")print(df[['fare']].head())print("Variance of Original Fare:", df['fare'].var())
This code snippet loads the dataset, outputs the first few entries of the 'fare' column, and prints the variance of the original 'fare'. Variance is a measure of how much the values in a dataset differ from the mean. It provides insight into the data's spread; a high variance indicates that the data points are spread out over a wider range of values. The formula for variance is:
σ2=N1i=1∑N(xi−μ)2
Where σ2 is the variance, N is the number of data points, xi represents each data point, and μ is the mean of the data. Observing its distribution and variance is crucial to deciding the appropriate transformation technique for normalization.
Original Fare: fare0 7.25001 71.28332 7.92503 53.10004 8.0500Variance of Original Fare: 2469.436845743116
The output reveals a high variance in the original 'fare', highlighting the need for transformation to manage spread and skewness.
Identifying skewness or wide spread in the 'fare' column prompts us to apply a log transformation. Log transformation is effective in compressing high-range values and stabilizing variance. It works by applying the natural logarithm (ln) to each data point, which can help in managing skewness and making the data more normally distributed. The formula for a log transformation is:
y=ln(x)
In this context, we use np.log1p() instead of just np.log() because np.log1p() computes the natural logarithm of (x + 1), represented as:
y=ln(x+1)
This approach is beneficial for handling small values and avoiding issues when x = 0, as the logarithm of zero is undefined.
Python
# Apply log transformation to faredf['fare_log'] = np.log1p(df['fare'])print("\nOriginal Fare and Log Transformed Fare:")print(df[['fare', 'fare_log']].head())print("Variance of Log Transformed Fare:", df['fare_log'].var())
Using np.log1p(), we achieve a logarithmic transformation of the 'fare' values. This adjustment mitigates skewness and reduces variance, making features more amenable to modeling.
text
Original Fare and Log Transformed Fare: fare fare_log0 7.2500 2.1102131 71.2833 4.2805932 7.9250 2.1888563 53.1000 3.9908344 8.0500 2.202765Variance of Log Transformed Fare: 0.9390545498271524
By significantly reducing variance, the log transformation offers a more normalized dataset, facilitating better pattern visibility.
While log transformation addresses high variance effectively, square root transformation offers a moderate approach, suitable for less skewed data. The square root transformation is applied using the formula y=x, which helps in reducing skewness by transforming each data point to its square root, providing a balanced transformation impact.
Python
# Apply square root transformation to faredf['fare_sqrt'] = np.sqrt(df['fare'])print("\nOriginal Fare and Square Root Transformed Fare:")print(df[['fare', 'fare_sqrt']].head())print("Variance of Square Root Transformed Fare:", df['fare_sqrt'].var())
The square root transformation provides another layer of adjustment by softening moderate skewness, giving a balanced transformation impact.
text
Original Fare and Square Root Transformed Fare: fare fare_sqrt0 7.2500 2.6925821 71.2833 8.4429442 7.9250 2.8151383 53.1000 7.2869754 8.0500 2.837252Variance of Square Root Transformed Fare: 8.679618578881051
The variance decreases moderately, confirming the transformation’s suitability for moderately skewed data.
To examine a less aggressive transformation, the cube root transformation offers a gentle normalization that can balance between preserving data characteristics and achieving normalization. The cube root transformation is applied using the formula y=3x, which smooths skewness by transforming each data point to its cube root, making it versatile for various modeling contexts.
Python
# Apply cube root transformation to faredf['fare_cbrt'] = np.cbrt(df['fare'])print("\nOriginal Fare and Cube Root Transformed Fare:")print(df[['fare', 'fare_cbrt']].head())print("Variance of Cube Root Transformed Fare:", df['fare_cbrt'].var())
Applying np.cbrt(), this transformation retains much of the original data's spread while smoothing skewness, making it versatile for various modeling contexts.
text
Original Fare and Cube Root Transformed Fare: fare fare_cbrt0 7.2500 1.9354381 71.2833 4.1463182 7.9250 1.9937303 53.1000 3.7586474 8.0500 2.004158Variance of Cube Root Transformed Fare: 1.1502270097208227
The modest reduction in variance demonstrates the cube root transformation’s ability to subtly normalize data.
Python
import pandas as pdimport numpy as np# Load the datasetdf = pd.read_csv("titanic.csv")# Original fareprint("Original Fare:")print(df[['fare']].head())print("Variance of Original Fare:", df['fare'].var())
text
Original Fare: fare0 7.25001 71.28332 7.92503 53.10004 8.0500Variance of Original Fare: 2469.436845743116
