PredictHealth's Model Validation Strategy: Data Splitting, Evaluation, and Visualization

Introduction And Lesson Overview

Welcome back! In the previous lesson, you learned how to clean and prepare PredictHealth’s customer database, ensuring your data is reliable and ready for modeling. You now understand how to handle missing values, treat outliers, normalize features, and encode categorical variables. With a clean dataset in hand, you are ready to take the next step: building, validating, and evaluating predictive models in a way that ensures they perform well not just on your current data, but also on new, unseen data. In this lesson, you will learn about PredictHealth’s model validation strategy . You will discover why it is important to split your data into different subsets, how to build a robust modeling pipeline, and how to evaluate your model’s performance using a variety of metrics and visualizations. By the end of this lesson, you will be able to confidently train, validate, and test your models, ensuring they are both accurate and reliable. You will also get a first look at cross-validation, a powerful technique for making the most of your data. Let’s get started!

Understanding Data Splitting

Before you train a predictive model, it is essential to split your data into separate parts. This is not just a formality — it is a best practice that helps you build models that generalize well to new data. You may remember this idea from earlier lessons, but now we will go deeper and use three distinct subsets: the training set, the validation set, and the test set. The training set is used to fit the model. The validation set helps you tune the model and check its performance during development, allowing you to make adjustments without biasing your final results. The test set is kept completely separate until the very end, providing an unbiased evaluation of your model’s true performance on unseen data. In practice, a common split is 70% for training, 15% for validation, and 15% for testing. Here is how you can do this using scikit-learn’s train_test_split function: Pythonfrom sklearn.model_selection import train_test_split # Assume X and y are your features and target variable X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, random_state=42) X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42) print(f"Training set size: {X_train.shape[0]}") print(f"Validation set size: {X_val.shape[0]}") print(f"Test set size: {X_test.shape[0]}") The output might look like this: plaintextTraining set size: 936 Validation set size: 201 Test set size: 201 This approach ensures that each subset serves its purpose, helping you avoid overfitting and giving you a realistic sense of how your model will perform in the real world.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Training The Model And Validating Performance

With your pipeline ready, you can now train your model using the training data. After training, you will evaluate its performance on the validation set. This step helps you understand how well your model is learning and whether it is likely to generalize to new data.

To train the model, simply call the fit method on your pipeline with the training data. Then, use the predict method to generate predictions on the validation set. You can measure performance using several metrics:

Mean Squared Error (MSE): The average squared difference between actual and predicted values.
Root Mean Squared Error (RMSE): The square root of MSE, which brings the error back to the original units.
Mean Absolute Error (MAE): The average absolute difference between actual and predicted values.
R-squared (R²): The proportion of variance in the target variable explained by the model.

Here is how you can do this in code:

Python

from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
import numpy as np

# Train the model
model_pipeline.fit(X_train, y_train)

# Predict on validation set
y_val_pred = model_pipeline.predict(X_val)

# Calculate metrics
val_mse = mean_squared_error(y_val, y_val_pred)
val_rmse = np.sqrt(val_mse)
val_mae = mean_absolute_error(y_val, y_val_pred)
val_r2 = r2_score(y_val, y_val_pred)

print("\nValidation Set Performance:")
print(f"Mean Squared Error (MSE): {val_mse:.2f}")
print(f"Root Mean Squared Error (RMSE): {val_rmse:.2f}")
print(f"Mean Absolute Error (MAE): {val_mae:.2f}")
print(f"R-squared (R²): {val_r2:.2f}")

from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
import numpy as np

# Train the model
model_pipeline.fit(X_train, y_train)

# Predict on validation set
y_val_pred = model_pipeline.predict(X_val)

# Calculate metrics
val_mse = mean_squared_error(y_val, y_val_pred)
val_rmse = np.sqrt(val_mse)
val_mae = mean_absolute_error(y_val, y_val_pred)
val_r2 = r2_score(y_val, y_val_pred)

print("\nValidation Set Performance:")
print(f"Mean Squared Error (MSE): {val_mse:.2f}")
print(f"Root Mean Squared Error (RMSE): {val_rmse:.2f}")
print(f"Mean Absolute Error (MAE): {val_mae:.2f}")
print(f"R-squared (R²): {val_r2:.2f}")

A sample output might be:

plaintext

Validation Set Performance:
Mean Squared Error (MSE): 32000000.00
Root Mean Squared Error (RMSE): 5656.85
Mean Absolute Error (MAE): 4200.12
R-squared (R²): 0.78

Validation Set Performance:
Mean Squared Error (MSE): 32000000.00
Root Mean Squared Error (RMSE): 5656.85
Mean Absolute Error (MAE): 4200.12
R-squared (R²): 0.78

These metrics give you a clear sense of how well your model is performing on data it has not seen during training.

Cross-Validation Technique

While splitting your data into training, validation, and test sets is a strong approach, sometimes you want an even more reliable estimate of your model's performance. This is where cross-validation comes in. Cross-validation involves splitting your data into several folds, training and testing the model on different combinations, and averaging the results.

When should you use each approach? If you have abundant data (typically thousands of samples), the three-way split (train/validation/test) is often preferred because it gives you a dedicated test set that remains completely untouched until final evaluation. However, when data is scarce, k-fold cross-validation makes better use of your limited samples by ensuring every data point gets used for both training and validation.

Cross-validation is particularly useful for model selection — comparing different algorithms or tuning hyperparameters — because it gives you a more stable estimate of performance across multiple data splits. However, it comes with a computational cost: you are essentially training your model k times instead of once, which can be significant for complex models or large datasets. For final model evaluation, many practitioners still prefer a held-out test set that has never been used in any training or validation process.

A common method is 5-fold cross-validation, where the data is split into five parts. Each part gets a turn as the validation set, while the others are used for training. This helps you see how your model performs across different subsets of the data.

In scikit-learn, you can use the cross_val_score function to do this easily. Here is an example:

Python

from sklearn.model_selection import cross_val_score

cv_scores = cross_val_score(model_pipeline, X, y, cv=5, scoring='neg_mean_squared_error')
cv_rmse_scores = np.sqrt(-cv_scores)

print("\nCross-Validation Results (RMSE):")
print(f"Mean RMSE: {cv_rmse_scores.mean():.2f}")
print(f"Standard Deviation: {cv_rmse_scores.std():.2f}")
print(f"Individual fold scores: {cv_rmse_scores}")

from sklearn.model_selection import cross_val_score

cv_scores = cross_val_score(model_pipeline, X, y, cv=5, scoring='neg_mean_squared_error')
cv_rmse_scores = np.sqrt(-cv_scores)

print("\nCross-Validation Results (RMSE):")
print(f"Mean RMSE: {cv_rmse_scores.mean():.2f}")
print(f"Standard Deviation: {cv_rmse_scores.std():.2f}")
print(f"Individual fold scores: {cv_rmse_scores}")

The output might be:

plaintext

Cross-Validation Results (RMSE):
Mean RMSE: 5700.23
Standard Deviation: 300.12
Individual fold scores: [5600.12 5900.45 5500.67 5800.34 5700.56]

Cross-Validation Results (RMSE):
Mean RMSE: 5700.23
Standard Deviation: 300.12
Individual fold scores: [5600.12 5900.45 5500.67 5800.34 5700.56]

Cross-validation gives you a more robust sense of your model's stability and helps you avoid being misled by a lucky or unlucky split.

Visualizing Model Performance

Numbers are important, but sometimes a picture is worth a thousand words. Visualizing your model's predictions can help you spot patterns, strengths, and weaknesses that metrics alone might miss.

One useful plot is the actual vs. predicted scatter plot. This shows how close your model's predictions are to the true values. Ideally, the points should fall along a diagonal line.

Here is how you can create this plot:

Python

import matplotlib.pyplot as plt

plt.figure(figsize=(10, 6))
plt.scatter(y_test, y_test_pred, alpha=0.5)
plt.plot([y.min(), y.max()], [y.min(), y.max()], 'k--', lw=2)
plt.xlabel('Actual Insurance Charges ($)')
plt.ylabel('Predicted Insurance Charges ($)')
plt.title('Actual vs. Predicted Insurance Charges (Test Set)')
plt.show()

import matplotlib.pyplot as plt

plt.figure(figsize=(10, 6))
plt.scatter(y_test, y_test_pred, alpha=0.5)
plt.plot([y.min(), y.max()], [y.min(), y.max()], 'k--', lw=2)
plt.xlabel('Actual Insurance Charges ($)')
plt.ylabel('Predicted Insurance Charges ($)')
plt.title('Actual vs. Predicted Insurance Charges (Test Set)')
plt.show()

You will see a scatter plot where each point represents a prediction. The closer the points are to the dashed line, the better your model is performing.

Another helpful visualization is the residual plot, which shows the difference between actual and predicted values (the residuals) against the predicted values. This can help you spot patterns that suggest your model is missing something.

Here is the code:

Python

plt.figure(figsize=(10, 6))
residuals = y_test - y_test_pred
plt.scatter(y_test_pred, residuals, alpha=0.5)
plt.axhline(y=0, color='red', linestyle='--')
plt.xlabel('Predicted Insurance Charges ($)')
plt.ylabel('Residuals ($)')
plt.title('Residual Plot')
plt.show()

plt.figure(figsize=(10, 6))
residuals = y_test - y_test_pred
plt.scatter(y_test_pred, residuals, alpha=0.5)
plt.axhline(y=0, color='red', linestyle='--')
plt.xlabel('Predicted Insurance Charges ($)')
plt.ylabel('Residuals ($)')
plt.title('Residual Plot')
plt.show()

If the residuals are randomly scattered around zero, it means your model's errors are evenly distributed, which is a good sign.

In the residual plot above, you can see a typical example of how residuals should look for a well-performing model. The points are scattered roughly evenly around the horizontal red line (which represents zero error), without showing any clear patterns or trends. This random distribution of residuals indicates that your model's errors are not systematically biased in any particular direction.

Notice that most residuals fall within a reasonable range around zero, though there are a few outliers with larger positive and negative residuals. This is normal and expected. If you saw a clear pattern in the residuals — such as a curve, a funnel shape, or clustering — it would suggest that your model is missing some important relationships in the data that could be captured with additional features or a different modeling approach.

Introduction And Lesson Overview

Understanding Data Splitting

Join the 1M+ learners on CodeSignal

Building The Preprocessing And Modeling Pipeline

Training The Model And Validating Performance

Final Model Evaluation On The Test Set

Cross-Validation Technique

Visualizing Model Performance

Summary And Preparation For Hands-On Practice