Implementing the Naive Bayes Classifier from Scratch in C++
Introduction
Welcome to our exploration tour of the Naive Bayes Classifier! This robust classification algorithm is renowned for its simplicity and effectiveness. We will implement it from scratch in C++, allowing you to leverage its sheer power without the need for any prebuilt libraries. Let's get started!
Recall
Let's do a quick recall of probability theory.
P(A) usually denotes the likelihood of a certain event A occurring. P(A∣B), on the other hand, indicates the probability of event A taking place, assuming event B has already happened.
For instance, let's imagine there's a bag housing three marbles - one red and two blue. Denote A as the event where a red marble is picked, and B when a blue one is drawn.
The probability of A, P(A), is 1/3 in this case.
Now, let's consider a scenario where a blue marble has been already drawn from the bag. This leaves us with one red and one blue marble in the bag. The probability of drawing a red marble (event A), given that a blue marble has already been extracted (event B), is denoted by P(A∣B). In this case, P(A∣B) would be 1/2, highlighting a higher likelihood of drawing a red marble following the initial removal of a blue one.
The Principle of Naive Bayes
Deriving the Naive Bayes Classifier Algorithm
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
The Naive Bayes algorithms rely on the Bayes' theorem. Let's recall it quickly. This theorem calculates the probability of an event based on prior knowledge of potentially related events. It is represented mathematically as:
P(A∣B)=P(B)P(B∣A)P(A)
Where P(A∣B) is the posterior probability of class (A) given predictor (B). It's what we are trying to calculate. P(B∣A) is the likelihood, which is the probability of the predictor given a class. P(B) is the marginal probability of predictor, and P(A) is the prior probability of the class. This formula forms the backbone of the Naive Bayes classifier.
The term 'naive' refers to the assumption that all variables in a dataset are independent of each other, which may not always be the case in real-life data. Nonetheless, it still offers robust performance and can be easily implemented.
In the context of machine learning, the Naive Bayes Classifier uses the Bayes theorem to compute the posterior probability of a class given a set of features and then classifies the outcome based on the highest posterior probability.
Assuming a binary class variable Y (binary means it can be equal to either 0 or 1) and features X1,X2,...,Xn, our task is to compute the posterior probability P(Y=1∣X1=x1,X2=x2,...,Xn=xn). By shedding the denominator from Bayes' theorem (since it doesn't depend on Y and is constant for all classes), we are left with the task of maximizing the probability of Y and X happening together P(Y,X)=P(X∣Y)P(Y), which forms the basis for Naive Bayes classification.
Example: Calculating Prior Probabilities
Example: Calculating Likelihoods
Implementing Naive Bayes Classifier
We approach the implementation of the Naive Bayes Classifier by first calculating the prior probabilities of each class, and then the likelihood of each feature given a class:
#include <iostream>#include <vector>#include <map>#include <string>#include <algorithm>using namespace std;struct DataPoint { map<string, string> features; string label; DataPoint(const map<string, string>& f, const string& l) : features(f), label(l) {}};map<string, double> calculate_prior_probabilities(const vector<string>& labels) { map<string, int> class_counts; int total_count = labels.size(); // Count occurrences of each class for (const string& label : labels) { class_counts[label]++; } // Calculate prior probabilities map<string, double> priors; for (const auto& pair : class_counts) { priors[pair.first] = static_cast<double>(pair.second) / total_count; } return priors;}map<string, map<string, map<string, double>>> calculate_likelihoods( const vector<DataPoint>& data, const vector<string>& feature_names) { map<string, map<string, map<string, double>>> likelihoods; // Get unique classes vector<string> classes; for (const DataPoint& point : data) { if (find(classes.begin(), classes.end(), point.label) == classes.end()) { classes.push_back(point.label); } } for (const string& feature : feature_names) { likelihoods[feature] = map<string, map<string, double>>(); for (const string& class_ : classes) { likelihoods[feature][class_] = map<string, double>(); // Count feature values for this class map<string, int> feature_counts; int class_count = 0; for (const DataPoint& point : data) { if (point.label == class_) { feature_counts[point.features.at(feature)]++; class_count++; } } // Calculate likelihoods for (const auto& feature_pair : feature_counts) { likelihoods[feature][class_][feature_pair.first] = static_cast<double>(feature_pair.second) / class_count; } } } return likelihoods;}
Armed with these utility functions, we can implement the Naive Bayes Classifier function:
vector<string> naive_bayes_classifier( const vector<DataPoint>& test_data, const map<string, double>& priors, const map<string, map<string, map<string, double>>>& likelihoods, const vector<string>& feature_names) { vector<string> predictions; for (const DataPoint& data_point : test_data) { map<string, double> class_probabilities; for (const auto& prior_pair : priors) { string class_ = prior_pair.first; class_probabilities[class_] = prior_pair.second; for (const string& feature : feature_names) { string feature_value = data_point.features.at(feature); // Get likelihood with default value for unseen features double likelihood = 0.0; if (likelihoods.at(feature).at(class_).find(feature_value) != likelihoods.at(feature).at(class_).end()) { likelihood = likelihoods.at(feature).at(class_).at(feature_value); } else { // Default probability for unseen values likelihood = 1.0 / (likelihoods.at(feature).at(class_).size() + 1); } class_probabilities[class_] *= likelihood; } } // Predict class with maximum posterior probability string predicted_class = class_probabilities.begin()->first; double max_prob = class_probabilities.begin()->second; for (const auto& prob_pair : class_probabilities) { if (prob_pair.second > max_prob) { max_prob = prob_pair.second; predicted_class = prob_pair.first; } } predictions.push_back(predicted_class); } return predictions;}
Understanding and Handling Data Issues in Naive Bayes
A recurring challenge in Naive Bayes is the handling of zero probabilities, i.e., when a category does not appear in the training data for a given class, resulting in a zero probability for that category. A known fix for this problem is applying Laplace or Add-1 smoothing, which adds a '1' to each category count to circumvent zero probabilities.
To apply Laplace smoothing, you only need to modify the likelihood calculation part of the calculate_likelihoods function. Here is the updated section with Laplace smoothing (the rest of the function remains unchanged):
This is indeed Laplace smoothing (also called Add-1 smoothing):
The numerator is increased by 1 for each feature value count.
The denominator is increased by the number of unique feature values for that feature.
You only need to replace the likelihood calculation loop in your original calculate_likelihoods function with the code above to enable Laplace smoothing. The rest of your implementation remains the same.
Using Naive Bayes Classifier
Here is a short example of predicting weather with our classifier:
The Naive Bayes Classifier predicts a class label based on the observed features. Owing to its simplicity, power, and speed, this classifier lends itself to challenging scenarios, including text classification, spam detection, and sentiment analysis.
Lesson Summary and Practice
Superb work! You've mastered the essentials of the Naive Bayes Classifier, from understanding its theory to crafting a Naive Bayes Classifier from scratch. The next phase is practice, which will consolidate your newly acquired skills. Enjoy the hands-on exercises lined up next. Delve deeper into your machine learning journey with the forthcoming lessons!
