Welcome back! In the previous lessons, you learned how to set up your SageMaker environment, upload your data to S3, and launch a training job using the SKLearn estimator. You also saw how SageMaker manages the training process and saves your trained model artifacts to S3. These are essential skills for any machine learning workflow in the cloud.

In this lesson, you will learn how to retrieve the most recent completed SageMaker training job, download the corresponding trained model from S3, and evaluate its performance on test data. You will find your latest training job, access the model artifacts, load the trained model, and test how well it performs on new data. This is a crucial step in any machine learning project, as it allows you to verify that your model is working as expected before moving on to deployment or further experimentation.

After running training jobs in SageMaker, you may want to access a specific job to download and evaluate its model. If you already know the exact training job name you want to work with (perhaps you noted it down when the job completed, or you're working with a specific job ID from your team), you can simply set it as a variable:

However, if you're working interactively and want to evaluate your most recent experiment, or if you're building an automated pipeline that needs to find the latest successful training job, you'll need to query SageMaker for this information. SageMaker provides a way to list all your training jobs and filter them by status.

To retrieve the most recent completed job programmatically, you use the list_training_jobs method. This method queries the SageMaker service and returns information about your training jobs. Let's break down how it works:

Let's understand each parameter:

• SortBy='CreationTime': This tells SageMaker to sort the jobs by when they were created
• SortOrder='Descending': This means the newest jobs will appear first (descending from newest to oldest)
• StatusEquals='Completed': This filters out any jobs that failed, stopped, or are still running - we only want successfully completed jobs
• NameContains='sagemaker-scikit-learn': This filters the results to only include jobs whose names contain the string , which helps narrow down to the specific SKLearn estimator training jobs you're interested in.

Once you have the training job name, the next step is to reconnect to that completed training job to access its information. When a SageMaker training job completes, it saves the model files to a specific S3 location, but you need a way to retrieve that location.

This is where the SKLearn.attach() method comes in. The attach() method allows you to "reconnect" to a training job that has already completed. Think of it as retrieving a reference to a past training job so you can access all its information and outputs. When you run a new training job, you have an estimator object that knows everything about that job. But when you come back later (maybe hours or days later), you need a way to recreate that connection - that's what attach() does.

Here's what happens when you use attach():

1. It connects to the SageMaker service and retrieves all the metadata about the specified training job
2. It reconstructs an estimator object with all the original settings and configurations
3. Most importantly, it retrieves the S3 location where the trained model was saved

When you attach to a training job, you're also able to see the status information from when that job was executed. For example, the completed training job would have gone through these stages:

These status messages show the lifecycle that the training job went through:

• Starting: SageMaker was setting up the computing resources
• Downloading: The Docker container with your training environment was being downloaded
• : Your model was actively being trained

Once you've successfully attached to a completed training job, you can access various properties of that job through the estimator object. One of the most important properties is the location where SageMaker saved your trained model.

The estimator.model_data property contains the S3 URI (Uniform Resource Identifier) where SageMaker automatically saved your model after training completed. A URI is like a web address, but for S3 storage - it tells us exactly where to find the file in Amazon's cloud storage.

Having this S3 location is crucial for the next step, where you'll download the model artifact to your local environment for evaluation.

Now that you have the S3 location, you need a way to download the file from S3 to your local environment. SageMaker provides a convenient tool called S3Downloader for this purpose. The S3Downloader is a utility class that simplifies downloading files from S3 - it handles authentication, network transfers, and error handling behind the scenes. Without it, you would need to write much more complex code using the lower-level AWS SDK.

Here's how to use it:

The S3Downloader.download() function takes two main parameters:

• The first parameter (model_s3_uri) is the source - where to download from
• The second parameter (".") is the destination - where to save the file locally (the dot means "current directory")

The downloaded file will be a compressed archive named model.tar.gz containing your trained model.

After downloading the model artifact, you will find that it is packaged as a compressed tarball (model.tar.gz). To use the trained model, you need to extract its contents and load the model file.

You can use Python's built-in tarfile module to extract the files. Once extracted, you will typically find a file named model.joblib, which contains your trained scikit-learn model. You can then use the joblib library to load this model into memory.

Here is how you can extract and load the model:

This code will make your trained model available as a Python object, ready for evaluation or inference.

When loading the model, you may see a warning if the scikit-learn version that was used to train your model in SageMaker is different from the version you have installed locally. This happens because the model remembers which version it was created with, and scikit-learn warns you when these don't match. For simple models like LinearRegression, this is generally safe to ignore and won't affect the model's ability to make predictions. However, when working with more complex models or different machine learning frameworks, it's important to be aware of version compatibility as significant changes between versions could affect model behavior.

To evaluate your model, you need to test it on data that it has not seen before. In this lesson, we will use our test dataset from the first course stored as a CSV file. The process involves loading the CSV file into a pandas DataFrame, then separating the features (input variables) from the target variable (the value you want to predict).

Here is how you can prepare your test data:

In this example, X_test contains all the features used for prediction, while y_test contains the actual house values. This separation is important for evaluating how well your model predicts unseen data.

With your model loaded and test data prepared, you can now evaluate the model's performance. This involves making predictions on the test set and comparing them to the actual values. Two common metrics for regression tasks are the R² score (coefficient of determination) and the Root Mean Squared Error (RMSE).

The R² score measures how well the model explains the variance in the target variable, with a value closer to 1 indicating better performance. The RMSE gives you an idea of the average prediction error in the same units as the target variable.

Here is how you can evaluate your model:

A sample output might look like this:

These metrics indicate that your model was successfully trained and is making reasonable predictions on new, unseen data. An R² score of 0.6456 means the model explains about 64.56% of the variance in house prices, while the RMSE of 0.6814 shows the model has learned meaningful patterns from the training data.

Beyond evaluating your model on a full test set, it is often useful to see how it performs on a single example. This can help you interpret the model's predictions and build confidence in its results.

To do this, we need to create a DataFrame with a single row of feature values and use the model's predict method.

A typical output might be:

This shows you the predicted house value for the given input features, making the model's output more tangible and easier to interpret.

In this lesson, you learned how to retrieve the most recent completed SageMaker training job, download the trained model artifacts from S3, extract and load the model, and evaluate its performance on test data. You also saw how to test the model with a single sample to better understand its predictions.

These steps are essential for validating your machine learning models and ensuring they perform well before moving on to deployment or further experimentation. You now have a complete workflow for training, retrieving, and evaluating models in SageMaker.

Next, you will have the opportunity to practice these skills with hands-on exercises. This will help reinforce what you have learned and prepare you for more advanced topics in machine learning with SageMaker. If this is your final unit, congratulations on reaching this milestone! Your ability to manage the full lifecycle of a machine learning model in the cloud is a valuable skill that will serve you well in your future projects.