Model Prediction and Evaluation

Welcome back! In the previous lesson, we learned how to train a simple machine learning model using the Linear SVM method with the caret package in R. Now that you have a trained model, it's time to make predictions and evaluate how well your model performs.

In this lesson, you will learn how to:

• Make predictions using your trained model.
• Evaluate the performance of your model using a confusion matrix.

Making predictions and evaluating your model are crucial steps in the machine learning workflow. They help you assess whether your model performs well on unseen data and identify areas for improvement.

In this lesson, we continue from where we left off in the last lesson. Before making predictions, remember that the initial steps include loading the dataset, splitting the data into training and testing sets, and training your model. Here’s a brief reminder of those steps:

# Load iris dataset
data(iris)

# For reproducibility
set.seed(123)

# Splitting data into train and test sets
trainIndex <- createDataPartition(iris$Species, p = 0.7, list = FALSE, times = 1)
irisTrain <- iris[trainIndex,]
irisTest  <- iris[-trainIndex,]

# Training a Linear SVM model
model <- train(Species ~ ., data = irisTrain, method = "svmLinear")

Once you have your trained model, the next step is to make predictions on your test data. In R, you can use the predict function from the caret package to generate these predictions. Here's how you can do it:

# Making predictions
predictions <- predict(model, irisTest)

In this code snippet:

• model is your trained machine learning model.
• irisTest is your test dataset, which contains the same features as your training data, including the target labels. The predict function will ignore the labels and use only the features for making predictions.

After making predictions, you need to evaluate how well your model performed. One of the common ways to do this is by using a confusion matrix. But what exactly is a confusion matrix?

A confusion matrix is a table used to evaluate the performance of a classification model. It compares the actual target values with the predicted values and provides a detailed breakdown of your model's performance. The matrix includes the following terms:

• True Positives (TP): The number of correct positive predictions.
• True Negatives (TN): The number of correct negative predictions.
• False Positives (FP): The number of incorrect positive predictions.
• False Negatives (FN): The number of incorrect negative predictions.

The rows in a confusion matrix represent the actual classes, and the columns represent the predicted classes. This makes it easy to see where your model is making correct and incorrect classifications.

Here's how you can create and print a confusion matrix in R:

confusion <- confusionMatrix(predictions, irisTest$Species)
print(confusion)

In this code snippet:

• confusionMatrix is a function from the caret package that takes the predictions and the actual target values as arguments and returns a confusion matrix.
• irisTest$Species is the actual target values from your test dataset.

Here’s a rough idea of what you might see:

Confusion Matrix and Statistics

            Reference
Prediction   setosa versicolor virginica
  setosa         15          0         0
  versicolor      0         15         1
  virginica       0          0        14

Overall Statistics
                                          
               Accuracy : 0.9778          
                 95% CI : (0.8823, 0.9994)
    No Information Rate : 0.3333          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.9667          
                                          
 Mcnemar's Test P-Value : NA              

Statistics by Class:

                     Class: setosa Class: versicolor Class: virginica
Sensitivity                 1.0000            1.0000           0.9333
Specificity                 1.0000            0.9667           1.0000
Pos Pred Value              1.0000            0.9375           1.0000
Neg Pred Value              1.0000            1.0000           0.9677
Prevalence                  0.3333            0.3333           0.3333
Detection Rate              0.3333            0.3333           0.3111
Detection Prevalence        0.3333            0.3556           0.3111
Balanced Accuracy           1.0000            0.9833           0.9667