Welcome to this lesson on factorization machines, an important model in the realm of recommendation systems. Factorization machines, or FM, excel in capturing interactions between variables, making them a powerful tool for both regression and classification tasks. For instance, they can predict a rating (regression) or calculate the likelihood of a recommendation (classification).
Review of Dataset Preparation
Before we delve into the implementation of a factorization machine, let's briefly revisit the dataset preparation process from the previous lesson. Even though we won't repeat the entire code here, it's crucial to remember the structure we've established.
In the prior lesson, you learned how to load JSON files and create a user-item interaction matrix using dummy variables. Additionally, you enriched the dataset with auxiliary features like user preferences and genre similarity. These steps laid the groundwork for accurately predicting ratings in a recommendation system. Recall the importance of these preparatory steps as we move forward.
Theory Behind
Latent Vectors
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Factorization machines leverage interactions between variables by decomposing them into simpler, latent factors. Mathematically, the prediction for a factorization machine can be expressed as:
xi: The individual features of the input vector x.
⟨vi,vj⟩: The dot product between the latent vectors of two features, capturing their pairwise interaction.
Latent vectors are fundamental components in factorization machines used to capture complex pairwise interactions between features. Each column in the dataset is represented by a latent vector, and the interaction between different columns is determined by the dot product of these latent vectors.
Let's consider the dataset with the columns described:
The columns user1, user2, user3, item1, item2, item3 are one-hot encoded features indicating which user-item pair the row represents.
uf1, uf2 are additional user features (e.g., user attributes).
if1, if2 are additional item features (e.g., item attributes).
r is the rating given by the user to the item.
In the context of this dataset, each feature column has an associated latent vector, which is initialized randomly and learned during training. These latent vectors help to infer interactions between features that are not explicitly represented in the data.
For demonstration, let's consider a latent factor size of 2 for simplicity, though in practice, n_factors can be larger.
Feature
Latent Vector
user1
[vu1,1,vu1,2]
user2
[vu2,1,vu2,2]
user3
[vu3,1,vu3,2]
item1
[vi1,1,vi1,2]
item2
[vi2,1,vi2,2]
item3
[vi3,1,vi3,2]
uf1
[vuf1,1,vuf1,2]
uf2
[vuf2,1,vuf2,2]
if1
[vif1,1,vif1,2]
if2
[vif2,1,vif2,2]
For the row example: user2 item3 uf1 uf2 if1 if2 r, the active features are user2, item3, uf1, uf2, if1, and if2.
Dot Product Calculation: The interaction between any two features (e.g., user2 and item3) is captured by the dot product of their latent vectors.
⟨vu2,vi3⟩=vu2,1⋅vi3,1+vu2,2⋅vi3,2
For each pair of interacting features, this dot product evaluates how much they co-influence each other in predicting the rating.
Extending to All Feature Pairs: The interactions will be summed for all feature pairs where interactions are considered:
interaction_term=∑pairs⟨vfeaturei,vfeaturej⟩
Example Calculation: Given vu2=[0.5,0.3], vi3=[0.4,0.7], the dot product is:
⟨vu2,vi3⟩=0.5⋅0.4+0.3⋅0.7=0.2+0.21=0.41
Latent vectors "encode" features allowing compact representations of complex relationships beyond direct correlations. They effectively project user-item-feature information into a latent space, where their interactions can be learned and utilized for making predictions.
Thus, by updating these latent vectors through the fit method, the factorization machine fine-tunes its understanding of how different features interact, boosting its predictive performance.
Implementing the Factorization Machine Model: Part 1
Let's move on to the implementation of the factorization machine model. We'll break this into parts to ensure clarity.
First, let's define the __init__ method to initialize the required data.
In the __init__ method, we initialize several key parameters of the factorization machine.
n_factors: This defines the number of components in each latent vector. It represents the dimensionality of the latent space for each feature, capturing the complexity of interactions.
n_features: This is the total number of features in the dataset.
Together, n_factors and n_features define the dimensions of the interaction matrix V, which is of size (n_features, n_factors). Each row in this matrix corresponds to a feature, and each column corresponds to a component of the latent vector for that feature.
The learning_rate, epochs, and reg are hyperparameters governing the learning process. The w0 is the global bias, W stores linear coefficients for features, and V contains the interaction factors, initialized with small random values.
Gradient Descent
Implementing the Factorization Machine Model: Part 2
Next, we define the fit method that uses gradient descent to train the algorithm.
def fit(self, X, y): m, n = X.shape for epoch in range(self.epochs): for i in range(m): # Calculate linear terms by combining global bias and feature coefficients linear_terms = self.w0 + np.dot(X[i], self.W) # Calculate interaction terms using dot product of feature interactions interaction_term = sum( (np.dot(X[i], self.V[:, f]) ** 2 - np.dot(X[i] ** 2, self.V[:, f] ** 2)) / 2 for f in range(self.n_factors)) # Combine linear terms and interaction terms to get predictions predictions = linear_terms + interaction_term # Calculate prediction error err = predictions - y[i] # Update global bias using gradient descent self.w0 -= self.learning_rate * err # Update linear coefficients with regularization self.W -= self.learning_rate * (err * X[i] + self.reg * self.W) # Update interaction factors for each latent feature for f in range(self.n_factors): V_f = self.V[:, f] self.V[:, f] -= self.learning_rate * ( err * (X[i] @ V_f - X[i] ** 2 * V_f) / 2 + self.reg * V_f)
Initialize Parameters and Loop: We begin by determining the shape of the data (m, n) and iterating over each epoch to train the model.
Calculate Linear Terms: The linear terms are computed by summing the global bias w0 and the dot product of the feature coefficients W with the data instance X[i].
Calculate Interaction Terms: For each latent factor f, interaction terms are computed by taking the difference between the square of dot products and the dot product of squared terms, capturing feature interactions.
Compute Predictions and Error: Combine linear and interaction terms for predictions, then compute the error by subtracting actual ratings from predicted values.
Update Global Bias: The global bias is updated with the gradient of the error.
Update Linear Coefficients: Linear coefficients are adjusted using gradient descent, with regularization to prevent overfitting.
Update Interaction Factors: Each interaction factor is updated using the error and incorporates regularization to fine-tune learning of feature interactions. The gradient for interaction factors includes two parts:
The term (X[i] @ V_f) computes the dot product between the feature vector and the current latent vector, highlighting the current influence of all features on the interaction term.
The term (X[i] ** 2 * V_f) is the element-wise multiplication between the squared feature vector and the current latent vector, used to adjust for non-linearity and overfitting in interactions.
This dual consideration ensures that interactions are learned without inflating the error, especially with regularization.
Implementing the Factorization Machine Model: Part 3
Finally, we define the predict method that will use model's coefficients to make predictions.
def predict(self, X): m, n = X.shape y_pred = np.zeros(m) for i in range(m): # Calculate linear terms by combining global bias and feature coefficients linear_terms = self.w0 + np.dot(X[i], self.W) # Calculate interaction terms for each data instance interaction_term = sum( (np.dot(X[i], self.V[:, f]) ** 2 - np.dot(X[i] ** 2, self.V[:, f] ** 2)) / 2 for f in range(self.n_factors)) # Sum linear and interaction terms for prediction y_pred[i] = linear_terms + interaction_term return y_pred
Initialize Predictions Array: Start by creating an array to store predictions for each data instance.
Calculate Linear Terms: For each instance, compute linear terms by adding the global bias to the dot product of features with their coefficients.
Calculate Interaction Terms: Iterate over all latent factors to compute the interaction term for each instance.
Store Predictions: For each instance, sum the linear and interaction terms to calculate and store the predicted value.
Making Predictions and Evaluating Model Performance
Conclusion and Summary
In this lesson, we successfully implemented and evaluated a factorization machine model for recommendation systems. We've gone from initializing parameters, through training, to making predictions and evaluating performance. This concludes our exploration of factorization machines and marks the end of this course module.
Congratulations on completing the course! The skills you've acquired here form a strong foundation for building and understanding recommendation systems. Continue exploring other models and refine your expertise in this dynamic field. Well done!
Before explaining the fit method, let's briefly discuss gradient descent, a cornerstone optimization algorithm for training models. The principle behind gradient descent is to minimize a function by iteratively moving in the opposite direction of the gradient of the function. We move in the opposite direction because the gradient indicates the direction of the steepest increase, while we aim to minimize the loss function. In the context of our factorization machine, we adjust model parameters to minimize the error between predicted and actual outputs.
The gradient descent update rule for a parameter θ is given by:
θ:=θ−α∂θ∂J(θ)
Here, α is the learning rate, dictating the step size of each iteration, and J(θ) is the cost function measuring prediction error.
Python
def fit(self, X, y): m, n = X.shape for epoch in range(self.epochs): for i in range(m): # Calculate linear terms by combining global bias and feature coefficients linear_terms = self.w0 + np.dot(X[i], self.W) # Calculate interaction terms using dot product of feature interactions interaction_term = sum( (np.dot(X[i], self.V[:, f]) ** 2 - np.dot(X[i] ** 2, self.V[:, f] ** 2)) / 2 for f in range(self.n_factors)) # Combine linear terms and interaction terms to get predictions predictions = linear_terms + interaction_term # Calculate prediction error err = predictions - y[i] # Update global bias using gradient descent self.w0 -= self.learning_rate * err # Update linear coefficients with regularization self.W -= self.learning_rate * (err * X[i] + self.reg * self.W) # Update interaction factors for each latent feature for f in range(self.n_factors): V_f = self.V[:, f] self.V[:, f] -= self.learning_rate * ( err * (X[i] @ V_f - X[i] ** 2 * V_f) / 2 + self.reg * V_f)
Python
def predict(self, X): m, n = X.shape y_pred = np.zeros(m) for i in range(m): # Calculate linear terms by combining global bias and feature coefficients linear_terms = self.w0 + np.dot(X[i], self.W) # Calculate interaction terms for each data instance interaction_term = sum( (np.dot(X[i], self.V[:, f]) ** 2 - np.dot(X[i] ** 2, self.V[:, f] ** 2)) / 2 for f in range(self.n_factors)) # Sum linear and interaction terms for prediction y_pred[i] = linear_terms + interaction_term return y_pred
Now that the factorization machine is ready, let's see it in action.
from data_extraction import extract_datasetfrom sklearn.model_selection import train_test_split# Extract datasetdf = extract_dataset()# Split dataset into train and testX = df.drop(columns=['rating']).valuesy = df['rating'].valuesX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)# Define and train factorization machine modelfm_model = SimpleFactorizationMachine(n_factors=3, n_features=X_train.shape[1], learning_rate=0.001, epochs=1000)fm_model.fit(X_train, y_train)# Test the modely_pred = fm_model.predict(X_test)# Evaluate model using Mean Absolute Errormae = np.mean(abs((y_test - y_pred)))print(f'Mean Absolute Error: {mae:.4f}')# Mean Absolute Error: 0.9364
The Mean Absolute Error (MAE) is defined as:
MAE=n1∑i=1n∣yi−y^i∣
where n is the number of observations, yi is the actual value, and y^i is the predicted value.
Here, we extract and split the dataset into training and testing partitions, initialize the factorization machine, and fit it with training data. Predictions are made on the test dataset, and we evaluate accuracy using the Mean Absolute Error (MAE). On small datasets, the accuracy may be limited, but larger datasets typically result in better performance.
Python
from data_extraction import extract_datasetfrom sklearn.model_selection import train_test_split# Extract datasetdf = extract_dataset()# Split dataset into train and testX = df.drop(columns=['rating']).valuesy = df['rating'].valuesX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)# Define and train factorization machine modelfm_model = SimpleFactorizationMachine(n_factors=3, n_features=X_train.shape[1], learning_rate=0.001, epochs=1000)fm_model.fit(X_train, y_train)# Test the modely_pred = fm_model.predict(X_test)# Evaluate model using Mean Absolute Errormae = np.mean(abs((y_test - y_pred)))print(f'Mean Absolute Error: {mae:.4f}')# Mean Absolute Error: 0.9364
