Tuning Autoencoders in R

Welcome! Today, we will explore the art of fine-tuning Autoencoders. Previously, we learned about Autoencoders and their usefulness in dimensionality reduction. Now, we turn our attention to Hyperparameters — adjustable settings that can optimize model performance. We will experiment with different network architectures (by changing layers and activation functions) and training parameters (such as learning rates and batch sizes) of an Autoencoder using R and the keras3 package. Ready to begin this hands-on journey? Let’s get started!

Hyperparameters are adjustable settings that influence how a machine learning model learns. They are generally divided into two types: architectural and learning hyperparameters. Architectural hyperparameters include elements like the number of hidden layers and units in a neural network. Learning hyperparameters include the learning rate, number of epochs, and batch size. Tuning these hyperparameters is essential for managing model complexity and improving performance.

Architectural hyperparameters define the structure of a neural network, such as the number of layers and units. Layers are computational components that transform input data, and units (neurons) produce activations. Let’s modify our Autoencoder and experiment with different activation functions using R and keras3:

library(keras3)

# Set random seed for reproducibility
set.seed(42)

# Function to create an autoencoder model
create_autoencoder <- function(input_dim, encoded_dim, learning_rate) {
  input_layer <- layer_input(shape = input_dim)
  # Add a dense layer with 'relu' activation
  encoded <- input_layer |> 
    layer_dense(units = encoded_dim, activation = "relu")
  # Add a dense output layer with 'sigmoid' activation
  decoded <- encoded |> 
    layer_dense(units = input_dim, activation = "sigmoid")
  
  autoencoder <- keras_model(input_layer, decoded)
  autoencoder |> compile(
    optimizer = optimizer_adam(learning_rate = learning_rate),
    loss = "mean_squared_error"
  )
  return(autoencoder)
}

Learning hyperparameters, such as learning rate and batch size, have a significant impact on training. Let’s see how changing these values affects our Autoencoder.

# Simulate training and testing data
x_train <- matrix(runif(1000 * 20), nrow = 1000, ncol = 20)
x_test <- matrix(runif(300 * 20), nrow = 300, ncol = 20)

# Training with a higher learning rate
learning_rate_fast <- 0.1
autoencoder_fast <- create_autoencoder(20, 16, learning_rate_fast)
history_fast <- autoencoder_fast |> fit(
  x = x_train, y = x_train,
  epochs = 50,
  batch_size = 256,
  shuffle = TRUE,
  validation_data = list(x_test, x_test),
  verbose = 0
)

Now, let’s train the same architecture with a slower learning rate and compare the results.

# Training with a slower learning rate
learning_rate_slow <- 0.01
autoencoder_slow <- create_autoencoder(20, 16, learning_rate_slow)
history_slow <- autoencoder_slow |> fit(
  x = x_train, y = x_train,
  epochs = 50,
  batch_size = 256,
  shuffle = TRUE,
  validation_data = list(x_test, x_test),
  verbose = 0
)