Comprehensive Analysis With Multiple Techniques: Part 2

Lesson Introduction

Welcome to our lesson on integrating multiple techniques for comprehensive data analysis! Today, we'll dive deep into the Titanic dataset, using powerful functions and methods from pandas and numpy to uncover valuable insights. The goal is to learn how to combine techniques like groupby, merge, and pivot tables for thorough analysis. Integrating multiple techniques is like preparing a delicious meal: you combine several ingredients to create a rich, flavorful dish. Similarly, combining data analysis techniques helps extract deeper insights from data. Let's start stepping through our code!

Combining Groupby and Aggregation: Part 1

First, we'll group the data by class and sex and calculate the mean values. Grouping helps us understand patterns within subgroups. Pythonimport seaborn as sns import pandas as pd import numpy as np # Load the Titanic dataset titanic = sns.load_dataset('titanic') # Group by 'class' and 'sex' with observed=True to specify the exact behavior expected class_sex_grouping = titanic.groupby(['class', 'sex'], observed=True).agg({ 'survived': 'mean', # Mean survival rate 'fare': 'mean', # Mean fare 'age': ['mean', 'std'] # Mean and standard deviation of age }).reset_index()import seaborn as sns import pandas as pd import numpy as np # Load the Titanic dataset titanic = sns.load_dataset('titanic') # Group by 'class' and 'sex' with observed=True to specify the exact behavior expected class_sex_grouping = titanic.groupby(['class', 'sex'], observed=True).agg({ 'survived': 'mean', # Mean survival rate 'fare': 'mean', # Mean fare 'age': ['mean', 'std'] # Mean and standard deviation of age }).reset_index() Using reset_index here is necessary to convert the multi-level index (created by the groupby operation) back into regular columns of the DataFrame. Without resetting the index, the resulting DataFrame would have class and sex as index levels, which can complicate further data manipulation and readability

Combining Groupby and Aggregation: Part 2

After grouping, we'll simplify the multi-level columns for readability. Python # Simplify multi-level columns class_sex_grouping.columns = ['class', 'sex', 'survived_mean', 'fare_mean', 'age_mean', 'age_std'] print(class_sex_grouping) # Simplify multi-level columns class_sex_grouping.columns = ['class', 'sex', 'survived_mean', 'fare_mean', 'age_mean', 'age_std'] print(class_sex_grouping) Output: class sex survived_mean fare_mean age_mean age_std 0 First female 0.968085 106.125798 34.611765 13.612052 1 First male 0.368852 67.226127 41.281386 15.139570 2 Second female 0.921053 21.970121 28.722973 12.872702 3 Second male 0.157407 19.741782 30.740707 14.793894 4 Third female 0.500000 16.118810 21.750000 12.729964 5 Third male 0.135447 12.661633 26.507589 12.159514 class sex survived_mean fare_mean age_mean age_std 0 First female 0.968085 106.125798 34.611765 13.612052 1 First male 0.368852 67.226127 41.281386 15.139570 2 Second female 0.921053 21.970121 28.722973 12.872702 3 Second male 0.157407 19.741782 30.740707 14.793894 4 Third female 0.500000 16.118810 21.750000 12.729964 5 Third male 0.135447 12.661633 26.507589 12.159514 This tells us if first-class passengers had higher survival rates and fares compared to third-class passengers.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

import seaborn as sns
import pandas as pd
import numpy as np

# Load the Titanic dataset
titanic = sns.load_dataset('titanic')

# Group by 'class' and 'sex' with observed=True to specify the exact behavior expected
class_sex_grouping = titanic.groupby(['class', 'sex'], observed=True).agg({
    'survived': 'mean',  # Mean survival rate
    'fare': 'mean',      # Mean fare
    'age': ['mean', 'std']  # Mean and standard deviation of age
}).reset_index()

Python

# Simplify multi-level columns
class_sex_grouping.columns = ['class', 'sex', 'survived_mean', 'fare_mean', 'age_mean', 'age_std']

print(class_sex_grouping)

    class     sex  survived_mean   fare_mean   age_mean    age_std
0   First  female       0.968085  106.125798  34.611765  13.612052
1   First    male       0.368852   67.226127  41.281386  15.139570
2  Second  female       0.921053   21.970121  28.722973  12.872702
3  Second    male       0.157407   19.741782  30.740707  14.793894
4   Third  female       0.500000   16.118810  21.750000  12.729964
5   Third    male       0.135447   12.661633  26.507589  12.159514

Python

# Pivot table with observed=True for grouping to avoid FutureWarning
pivot_table = class_sex_grouping.pivot_table(
    index='class', 
    columns='sex', 
    values=['survived_mean', 'fare_mean', 'age_mean', 'age_std'],
    observed=True
)

print(pivot_table)
#              survived_mean                 
# sex               female      male      
# class                                                                                                          
# First          0.968085   0.368852  
# Second         0.921053   0.157407   
# Third          0.500000   0.135447   

#              fare_mean                    
# Analogous

#              age_mean                     
# Analogous

#              age_std                       
# Analogous

# Pivot table with observed=True for grouping to avoid FutureWarning
pivot_table = class_sex_grouping.pivot_table(
    index='class', 
    columns='sex', 
    values=['survived_mean', 'fare_mean', 'age_mean', 'age_std'],
    observed=True
)

print(pivot_table)
#              survived_mean                 
# sex               female      male      
# class                                                                                                          
# First          0.968085   0.368852  
# Second         0.921053   0.157407   
# Third          0.500000   0.135447   

#              fare_mean                    
# Analogous

#              age_mean                     
# Analogous

#              age_std                       
# Analogous

Python

# Adding a 'child' column: whether the passenger is a child (age < 18)
titanic['is_child'] = titanic['age'] < 18
print(titanic['is_child'])
# 0      False
# 1      False
# 2      False
# 3      False
# 4      False
# ...

# Adding a 'child' column: whether the passenger is a child (age < 18)
titanic['is_child'] = titanic['age'] < 18
print(titanic['is_child'])
# 0      False
# 1      False
# 2      False
# 3      False
# 4      False
# ...

Python

# Analyze survival rates by class and whether the passenger is a child or not
survival_by_class_child = titanic.pivot_table(
    'survived', index='class', columns='is_child', aggfunc='mean',
    observed=True
)

print(survival_by_class_child)
# is_child         False     True
# class                          
# First     0.612745  0.916667
# Second    0.409938  0.913043
# Third     0.217918  0.371795

# Analyze survival rates by class and whether the passenger is a child or not
survival_by_class_child = titanic.pivot_table(
    'survived', index='class', columns='is_child', aggfunc='mean',
    observed=True
)

print(survival_by_class_child)
# is_child         False     True
# class                          
# First     0.612745  0.916667
# Second    0.409938  0.913043
# Third     0.217918  0.371795

Python

# Merge this pivot table with the original grouped data for a comprehensive view
comprehensive_view = pd.merge(
    class_sex_grouping, 
    survival_by_class_child, 
    on='class', 
    how='left'
)

print(comprehensive_view)

# Merge this pivot table with the original grouped data for a comprehensive view
comprehensive_view = pd.merge(
    class_sex_grouping, 
    survival_by_class_child, 
    on='class', 
    how='left'
)

print(comprehensive_view)

    class     sex  survived_mean  ...    age_std     False      True
0   First  female       0.968085  ...  13.612052  0.612745  0.916667
1   First    male       0.368852  ...  15.139570  0.612745  0.916667
2  Second  female       0.921053  ...  12.872702  0.409938  0.913043
3  Second    male       0.157407  ...  14.793894  0.409938  0.913043
4   Third  female       0.500000  ...  12.729964  0.217918  0.371795
5   Third    male       0.135447  ...  12.159514  0.217918  0.371795

    class     sex  survived_mean  ...    age_std     False      True
0   First  female       0.968085  ...  13.612052  0.612745  0.916667
1   First    male       0.368852  ...  15.139570  0.612745  0.916667
2  Second  female       0.921053  ...  12.872702  0.409938  0.913043
3  Second    male       0.157407  ...  14.793894  0.409938  0.913043
4   Third  female       0.500000  ...  12.729964  0.217918  0.371795
5   Third    male       0.135447  ...  12.159514  0.217918  0.371795

Python

# Rename the columns for clarity
comprehensive_view.rename(columns={False: 'adult_survival_rate', True: 'child_survival_rate'}, inplace=True)

# Rename the columns for clarity
comprehensive_view.rename(columns={False: 'adult_survival_rate', True: 'child_survival_rate'}, inplace=True)

Python

# Pivot table with observed=True for grouping to avoid FutureWarning
pivot_table = class_sex_grouping.pivot_table(
    index='class', 
    columns='sex', 
    values=['survived_mean', 'fare_mean', 'age_mean', 'age_std'],
    observed=True
)

print(pivot_table)
#              survived_mean                 
# sex               female      male      
# class                                                                                                          
# First          0.968085   0.368852  
# Second         0.921053   0.157407   
# Third          0.500000   0.135447   

#              fare_mean                    
# Analogous

#              age_mean                     
# Analogous

#              age_std                       
# Analogous

Python

# Adding a 'child' column: whether the passenger is a child (age < 18)
titanic['is_child'] = titanic['age'] < 18
print(titanic['is_child'])
# 0      False
# 1      False
# 2      False
# 3      False
# 4      False
# ...

Python

# Analyze survival rates by class and whether the passenger is a child or not
survival_by_class_child = titanic.pivot_table(
    'survived', index='class', columns='is_child', aggfunc='mean',
    observed=True
)

print(survival_by_class_child)
# is_child         False     True
# class                          
# First     0.612745  0.916667
# Second    0.409938  0.913043
# Third     0.217918  0.371795

Python

# Merge this pivot table with the original grouped data for a comprehensive view
comprehensive_view = pd.merge(
    class_sex_grouping, 
    survival_by_class_child, 
    on='class', 
    how='left'
)

print(comprehensive_view)

    class     sex  survived_mean  ...    age_std     False      True
0   First  female       0.968085  ...  13.612052  0.612745  0.916667
1   First    male       0.368852  ...  15.139570  0.612745  0.916667
2  Second  female       0.921053  ...  12.872702  0.409938  0.913043
3  Second    male       0.157407  ...  14.793894  0.409938  0.913043
4   Third  female       0.500000  ...  12.729964  0.217918  0.371795
5   Third    male       0.135447  ...  12.159514  0.217918  0.371795

Python

# Rename the columns for clarity
comprehensive_view.rename(columns={False: 'adult_survival_rate', True: 'child_survival_rate'}, inplace=True)