Building and Evaluating a Model

Welcome back! You're now ready to build and evaluate machine learning models. You have learned how to preprocess the mtcars dataset and how to split the data into training and testing sets. Now, let's take it a step further and construct a logistic regression model.

In this lesson, you will:

1. Train a logistic regression model using the mtcars dataset.
2. Understand the importance of logistic regression in binary classification tasks.
3. Display and interpret model details to evaluate their performance.
4. Interpret warnings generated during model training and understand their implications.

By the end of this lesson, you will be able to:

• Build a logistic regression model using the caret library in R.
• Print and interpret the details of the model, including key performance metrics.
• Explain common warnings that may arise during model training and their significance.

Here's a key snippet of the code you'll be working with:

# Load the mtcars dataset
data(mtcars)

# Set seed for reproducibility
set.seed(123)

# Convert categorical columns to factors
mtcars$am <- as.factor(mtcars$am)
mtcars$cyl <- as.factor(mtcars$cyl)
mtcars$vs <- as.factor(mtcars$vs)
mtcars$gear <- as.factor(mtcars$gear)
mtcars$carb <- as.factor(mtcars$carb)

# Splitting data into training and testing sets
trainIndex <- createDataPartition(mtcars$am, p = 0.7, list = FALSE, times = 1)
trainData <- mtcars[trainIndex,]
testData <- mtcars[-trainIndex,]

# Feature scaling (excluding factor columns)
numericColumns <- sapply(trainData, is.numeric)
preProcValues <- preProcess(trainData[, numericColumns], method = c("center", "scale"))
trainData[, numericColumns] <- predict(preProcValues, trainData[, numericColumns])
testData[, numericColumns] <- predict(preProcValues, testData[, numericColumns])

# Train a logistic regression model, and display warnings
withCallingHandlers({
    model <- train(am ~ mpg + hp + wt, data = trainData, method = "glm", family = "binomial")
}, warning = function(w) {
    message("Warning: ", conditionMessage(w))
    invokeRestart("muffleWarning")
})

# Display the model details
print(model)

Let's understand the train function parameters in more depth:

• am ~ mpg + hp + wt: This formula specifies that we are trying to predict the am (transmission) column using mpg (miles per gallon), hp (horsepower), and wt (weight) as predictors.
• data = trainData: This specifies the dataset to be used for training the model.
• method = "glm": This indicates that we are using generalized linear models for training.
• family = "binomial": This specifies the family of the model, which in this case is binomial logistic regression since am is a binary outcome.

In the above code, we use withCallingHandlers to train the model and handle any warnings that might occur during the training process. The withCallingHandlers function allows us to catch warnings and handle them in a specific way, while still allowing the code to run. In this case, we are capturing warnings as messages and using invokeRestart("muffleWarning") to suppress them.

The output when displaying the model details is as follows:

Generalized Linear Model 

24 samples
 3 predictor
 2 classes: '0', '1' 

No pre-processing
Resampling: Bootstrapped (25 reps) 
Summary of sample sizes: 24, 24, 24, 24, 24, 24, ... 
Resampling results:

  Accuracy   Kappa    
  0.7824604  0.5547113

Note that the evaluation was performed on the training set using bootstrapped resampling, which is a technique that involves creating multiple training sets by randomly sampling the original data with replacement, and helps provide a more robust estimate of model performance by training the model multiple times on different variations of the data.

To understand the output, let's review the following performance metrics:

• Accuracy: This measures the proportion of correct predictions made by the model out of all predictions. For example, an accuracy of 0.78 means the model correctly predicted 78% of the cases.
• Kappa: This adjusts the accuracy to account for the possibility of the agreement occurring by chance. A Kappa value of 1 indicates perfect agreement, while 0 means the agreement is no better than random guessing.