Discovering Key Cost Factors: Correlation and Visualization in Insurance Data | CodeSignal Learn

Discovering Key Cost Factors: Correlation and Visualization in Insurance Data | CodeSignal Learn

Introduction & Lesson Overview

Welcome back! In the last lesson, you learned how to use different types of plots — such as histograms, bar charts, scatter plots, and boxplots — to explore the PredictHealth insurance dataset . These visualizations helped you spot patterns, compare groups, and get a general sense of how insurance charges vary across different customer profiles. In this lesson, we will take the next step: moving from simply visualizing the data to actually quantifying the relationships between customer attributes and insurance costs. Our main goal is to discover which factors — like age, bmi, number of children, and smoking status — most strongly influence insurance pricing. You will learn how to use correlation analysis and visualization techniques to identify these key cost drivers. By the end of this lesson, you will be able not only to see patterns but also to measure and compare the strength of these relationships, setting the stage for building predictive models in future lessons.

Preparing the Data for Analysis

Before we can analyze relationships, we need to make sure our data is ready. As a reminder, in previous lessons, you already checked for missing values and cleaned the dataset. Now, we will focus on selecting the most relevant numerical features for our analysis: age, bmi, children, and charges . These columns are important because they are either continuous or count data, making them suitable for correlation analysis. Here is how you can select these columns from the insurance dataset: Python numerical_data = insurance_data[['age', 'bmi', 'children', 'charges']] This line creates a new DataFrame called numerical_data that contains only the columns we want to analyze. By narrowing our focus to these features, we can more easily interpret the results and avoid confusion from unrelated variables.

Computing and Interpreting the Correlation Matrix

Now that we have our numerical features, we can calculate the correlation matrix. Correlation measures how strongly two variables move together. In Python, the .corr() method computes the Pearson correlation coefficient for each pair of columns. The result is a matrix where each value ranges from -1 (perfect negative correlation) to 1 (perfect positive correlation), with 0 meaning no linear relationship. Important Note : While correlation tells us about the strength and direction of relationships between variables, it's crucial to remember that correlation does not imply causation . Just because two variables are correlated doesn't mean one causes the other. For example, if age and charges are positively correlated, this doesn't necessarily mean that getting older directly causes higher insurance costs — there could be other factors at play, such as age-related health conditions. When working with insurance data, always consider that multiple factors might influence costs, and correlation analysis helps us identify relationships that warrant further investigation. Let's compute and display the correlation matrix: Python correlation_matrix = numerical_data.corr() print("Correlation Matrix:") print(correlation_matrix) The output will look something like this: Correlation Matrix: age bmi children charges age 1.000000 0.109272 0.042469 0.299009 bmi 0.109272 1.000000 0.012759 0.198341 children 0.042469 0.012759 1.000000 0.067998 charges 0.299009 0.198341 0.067998 1.000000 In this matrix, the diagonal values are always 1 because each variable is perfectly correlated with itself. The other values show the strength and direction of the relationship between each pair. For example, the correlation between age and charges is about 0.30, suggesting a moderate positive relationship: as age increases, charges tend to increase as well. The correlation between children and charges is much lower, indicating a weaker relationship.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Visualizing Correlations with a Heatmap

While the correlation matrix gives us the numbers, it can be hard to quickly spot the strongest relationships. A heatmap is a visual tool that uses color gradients to make these patterns stand out. In a heatmap, strong positive correlations are shown in one color (such as red), strong negative correlations in another (such as blue), and weak or no correlations in neutral colors. Here is how you can create a heatmap of the correlation matrix: plt.figure(figsize=(10, 8)) sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', vmin=-1, vmax=1) plt.title('Correlation Matrix of Numerical Features') plt.show() This code creates a large, easy-to-read heatmap. The annot=True argument displays the correlation values inside each cell, while the cmap='coolwarm' color scheme helps you quickly see which relationships are strong or weak. The vmin and vmax settings ensure the color scale is consistent from -1 to 1. On the heatmap you can notice that the cells with the highest positive values (closer to red) indicate stronger positive relationships, while those closer to blue indicate negative relationships. For example, the cell for age and charges are a lighter color, showing a moderate positive correlation, while the cell for children and charges are closer to white, indicating a weak relationship.

Exploring Relationships with Scatter Plots and Trend Lines

To better understand how individual features relate to insurance charges, we can use scatter plots with regression lines. These plots show each customer as a point, with the x-axis representing the feature (such as age or bmi) and the y-axis representing charges. The regression line (trend line) helps you see the overall direction of the relationship. Let’s look at two examples: age vs. charges and bmi vs. charges. plt.figure(figsize=(15, 10)) # Age vs. Charges plt.subplot(2, 2, 1) sns.regplot(x='age', y='charges', data=insurance_data, scatter_kws={'alpha':0.4}, line_kws={'color':'red'}) plt.title('Age vs. Charges') # BMI vs. Charges plt.subplot(2, 2, 2) sns.regplot(x='bmi', y='charges', data=insurance_data, scatter_kws={'alpha':0.4}, line_kws={'color':'red'}) plt.title('BMI vs. Charges') plt.tight_layout() plt.show() In the first plot, you can see that as age increases, insurance charges tend to rise, but there is still a lot of spread. The red regression line shows the general upward trend. In the second plot, you may notice a similar but weaker trend for bmi. Some outliers — such as customers with very high charges — may stand out, especially among those with high bmi. These visualizations help you see not just the strength of the relationship, but also the presence of outliers or clusters that might affect your analysis.

Analyzing Group-Level Trends with Boxplots

While scatter plots are great for continuous variables, boxplots are useful for comparing distributions across groups. For example, you might want to see how insurance charges differ by the number of children or by smoker status. Here is how you can create these boxplots: # Children vs. Charges plt.figure(figsize=(15, 5)) plt.subplot(1, 2, 1) sns.boxplot(x='children', y='charges', data=insurance_data, palette='Set3') plt.title('Number of Children vs. Charges') # Smoker vs. Charges plt.subplot(1, 2, 2) sns.boxplot(x='smoker', y='charges', data=insurance_data, palette='Set2') plt.title('Smoker Status vs. Charges') plt.tight_layout() plt.show() In the first boxplot, each different color represents a different number of children (0, 1, 2, 3, etc.). You can see separate boxes for customers with 0 children, 1 child, 2 children, and so on, with each category getting a distinct color from the 'Set3' palette. This makes it easy to compare the distribution of charges across different family sizes. You may notice that the median charges do not change much as the number of children increases, suggesting a weak relationship. The second boxplot compares smokers and non-smokers, with each group getting a different color from the 'Set2' palette. This plot usually shows a dramatic difference: smokers tend to have much higher charges, and the spread is also greater, indicating more variability in costs among smokers.

Comparing Group Averages with Summary Statistics

To get a clearer picture, you can also look at average charges by group using summary statistics: print("\nAverage charges by smoker status:") print(insurance_data.groupby('smoker')['charges'].mean()) print("\nAverage charges by region:") print(insurance_data.groupby('region')['charges'].mean()) print("\nAverage charges by gender:") print(insurance_data.groupby('sex')['charges'].mean()) The output might look like this: Average charges by smoker status: smoker no 8434.27 yes 32050.23 Name: charges, dtype: float64 Average charges by region: region northeast 13406.38 northwest 12417.58 southeast 14735.41 southwest 12346.94 Name: charges, dtype: float64 Average charges by gender: sex female 12569.58 male 13956.75 Name: charges, dtype: float64 These results show that smoking status has a very strong effect on charges, while region and gender have smaller differences. This kind of group-level analysis helps PredictHealth understand which customer segments are most costly and why.

Python

plt.figure(figsize=(15, 10))

# Age vs. Charges
plt.subplot(2, 2, 1)
sns.regplot(x='age', y='charges', data=insurance_data, scatter_kws={'alpha':0.4}, line_kws={'color':'red'})
plt.title('Age vs. Charges')

# BMI vs. Charges
plt.subplot(2, 2, 2)
sns.regplot(x='bmi', y='charges', data=insurance_data, scatter_kws={'alpha':0.4}, line_kws={'color':'red'})
plt.title('BMI vs. Charges')

plt.tight_layout()
plt.show()

Python

# Children vs. Charges
plt.figure(figsize=(15, 5))
plt.subplot(1, 2, 1)
sns.boxplot(x='children', y='charges', data=insurance_data, palette='Set3')
plt.title('Number of Children vs. Charges')

# Smoker vs. Charges
plt.subplot(1, 2, 2)
sns.boxplot(x='smoker', y='charges', data=insurance_data, palette='Set2')
plt.title('Smoker Status vs. Charges')

plt.tight_layout()
plt.show()

Python

print("\nAverage charges by smoker status:")
print(insurance_data.groupby('smoker')['charges'].mean())

print("\nAverage charges by region:")
print(insurance_data.groupby('region')['charges'].mean())

print("\nAverage charges by gender:")
print(insurance_data.groupby('sex')['charges'].mean())

Average charges by smoker status:
smoker
no     8434.27
yes   32050.23
Name: charges, dtype: float64

Average charges by region:
region
northeast    13406.38
northwest    12417.58
southeast    14735.41
southwest    12346.94
Name: charges, dtype: float64

Average charges by gender:
sex
female    12569.58
male      13956.75
Name: charges, dtype: float64