Implementing the Naive Bayes Classifier from Scratch in Python

Implementing the Naive Bayes Classifier from Scratch in Python | CodeSignal Learn

Introduction

Welcome to our exploration tour of the Naive Bayes Classifier! This robust classification algorithm is renowned for its simplicity and effectiveness. We will implement it from scratch in Python, allowing you to leverage its sheer power without the need for any prebuilt libraries. Let's get started!

Recall

P(A)

The Principle of Naive Bayes

P(A|B) = \frac{P(B|A)P(A)}{P(B)}

Deriving the Naive Bayes Classifier Algorithm

Y

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

The Naive Bayes algorithms rely on the Bayes' theorem. Let's recall it quickly. This theorem calculates the probability of an event based on prior knowledge of potentially related events. It is represented mathematically as:

$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$

Where $P(A|B)$ is the posterior probability of class ( $A$ ) given predictor ( $B$ ). It's what we are trying to calculate. $P(B|A)$ is the likelihood, which is the probability of the predictor given a class. $P(B)$ is the marginal probability of predictor, and $P(A)$ is the prior probability of the class. This formula forms the backbone of the Naive Bayes classifier.

The term 'naive' refers to the assumption that all variables in a dataset are independent of each other, which may not always be the case in real-life data. Nonetheless, it still offers robust performance and can be easily implemented.

In the context of machine learning, the Naive Bayes Classifier uses the Bayes theorem to compute the posterior probability of a class given a set of features and then classifies the outcome based on the highest posterior probability.

Assuming a binary class variable $Y$ (binary means it can be equal to either 0 or 1) and features $X_1, X_2, ..., X_n$ , our task is to compute the posterior probability $P(Y=1 | X_1=x_1, X_2=x_2,...,X_n=x_n)$ . By shedding the denominator from Bayes' theorem (since it doesn't depend on $Y$ and is constant for all classes), we are left with the task of maximizing the probability of $Y$ and $X$ happening together $P(Y, X) = P(X|Y)P(Y)$ , which forms the basis for Naive Bayes classification.

Example: Calculating Prior Probabilities

P(\text{Sunny}) = \frac{3}{7} \approx 0.43

Implementing Naive Bayes Classifier

We approach the implementation of the Naive Bayes Classifier by first calculating the prior probabilities of each class, and then the likelihood of each feature given a class: Pythonimport pandas as pd def calculate_prior_probabilities(y): # Calculate prior probabilities for each class return y.value_counts(normalize=True) def calculate_likelihoods(X, y): likelihoods = {} for column in X.columns: likelihoods[column] = {} for class_ in y.unique(): # Filter feature column data for each class class_data = X[y == class_][column] counts = class_data.value_counts() total_count = len(class_data) # Total count of instances for current class likelihoods[column][class_] = counts / total_count # Direct likelihoods without smoothing return likelihoodsimport pandas as pd def calculate_prior_probabilities(y): # Calculate prior probabilities for each class return y.value_counts(normalize=True) def calculate_likelihoods(X, y): likelihoods = {} for column in X.columns: likelihoods[column] = {} for class_ in y.unique(): # Filter feature column data for each class class_data = X[y == class_][column] counts = class_data.value_counts() total_count = len(class_data) # Total count of instances for current class likelihoods[column][class_] = counts / total_count # Direct likelihoods without smoothing return likelihoods Armed with these utility functions, we can implement the Naive Bayes Classifier function: Pythondef naive_bayes_classifier(X_test, priors, likelihoods): predictions = [] for _, data_point in X_test.iterrows(): class_probabilities = {} for class_ in priors.index: class_probabilities[class_] = priors[class_] for feature in X_test.columns: # Use .get to safely retrieve probability and get a default of 1/total to handle unseen values feature_probs = likelihoods[feature][class_] class_probabilities[class_] *= feature_probs.get(data_point[feature], 1 / (len(feature_probs) + 1)) # Predict class with maximum posterior probability predictions.append(max(class_probabilities, key=class_probabilities.get)) return predictionsdef naive_bayes_classifier(X_test, priors, likelihoods): predictions = [] for _, data_point in X_test.iterrows(): class_probabilities = {} for class_ in priors.index: class_probabilities[class_] = priors[class_] for feature in X_test.columns: # Use .get to safely retrieve probability and get a default of 1/total to handle unseen values feature_probs = likelihoods[feature][class_] class_probabilities[class_] *= feature_probs.get(data_point[feature], 1 / (len(feature_probs) + 1)) # Predict class with maximum posterior probability predictions.append(max(class_probabilities, key=class_probabilities.get)) return predictions

Understanding and Handling Data Issues in Naive Bayes

A recurring challenge in Naive Bayes is the handling of zero probabilities, i.e., when a category does not appear in the training data for a given class, resulting in a zero probability for that category. A known fix for this problem is applying Laplace or Add-1 smoothing, which adds a '1' to each category count to circumvent zero probabilities. You can integrate Laplace smoothing into the calculate_likelihoods function as follows: Pythondef calculate_likelihoods_with_smoothing(X, y): likelihoods = {} for column in X.columns: likelihoods[column] = {} for class_ in y.unique(): # Calculate normalized counts with smoothing class_data = X[y == class_][column] counts = class_data.value_counts() total_count = len(class_data) + len(X[column].unique()) # total count with smoothing likelihoods[column][class_] = (counts + 1) / total_count # add-1 smoothing return likelihoodsdef calculate_likelihoods_with_smoothing(X, y): likelihoods = {} for column in X.columns: likelihoods[column] = {} for class_ in y.unique(): # Calculate normalized counts with smoothing class_data = X[y == class_][column] counts = class_data.value_counts() total_count = len(class_data) + len(X[column].unique()) # total count with smoothing likelihoods[column][class_] = (counts + 1) / total_count # add-1 smoothing return likelihoods The numerator is increased by 1 and the denominator by the count of unique categories to accommodate the added 1's.

Using Naive Bayes Classifier

Here is a short example of predicting weather with our classifier: Pythondata = { 'Temperature': ['Hot', 'Hot', 'Cold', 'Hot', 'Cold', 'Cold', 'Cold'], 'Humidity': ['High', 'High', 'Normal', 'Normal', 'High', 'Normal', 'Normal'], 'Weather': ['Sunny', 'Sunny', 'Snowy', 'Rainy', 'Snowy', 'Snowy', 'Sunny'] } df = pd.DataFrame(data) # Split features and labels X = df[['Temperature', 'Humidity']] y = df['Weather'] # Calculate prior probabilities priors = calculate_prior_probabilities(y) # Calculate likelihoods with smoothing likelihoods = calculate_likelihoods_with_smoothing(X, y) # New observation X_test = pd.DataFrame([{'Temperature': 'Cold', 'Humidity': 'Normal'}]) # Make prediction prediction = naive_bayes_classifier(X_test, priors, likelihoods) print("Predicted Weather: ", prediction[0]) # Output: Predicted Weather: Snowydata = { 'Temperature': ['Hot', 'Hot', 'Cold', 'Hot', 'Cold', 'Cold', 'Cold'], 'Humidity': ['High', 'High', 'Normal', 'Normal', 'High', 'Normal', 'Normal'], 'Weather': ['Sunny', 'Sunny', 'Snowy', 'Rainy', 'Snowy', 'Snowy', 'Sunny'] } df = pd.DataFrame(data) # Split features and labels X = df[['Temperature', 'Humidity']] y = df['Weather'] # Calculate prior probabilities priors = calculate_prior_probabilities(y) # Calculate likelihoods with smoothing likelihoods = calculate_likelihoods_with_smoothing(X, y) # New observation X_test = pd.DataFrame([{'Temperature': 'Cold', 'Humidity': 'Normal'}]) # Make prediction prediction = naive_bayes_classifier(X_test, priors, likelihoods) print("Predicted Weather: ", prediction[0]) # Output: Predicted Weather: Snowy The Naive Bayes Classifier predicts a class label based on the observed features. Owing to its simplicity, power, and speed, this classifier lends itself to challenging scenarios, including text classification, spam detection, and sentiment analysis.

Consider a dataset with the class Weather having Sunny (3 instances), Rainy (1 instance), and Snowy (3 instances).

Temperature	Humidity	Weather
Hot	High	Sunny
Hot	High	Sunny
Cold	Normal	Snowy
Hot	Normal	Rainy
Cold	High	Snowy
Cold	Normal	Snowy
Cold	Normal	Sunny

The prior probabilities are calculated as follows:

$P(\text{Sunny}) = \frac{3}{7} \approx 0.43$
$P(\text{Rainy}) = \frac{1}{7} \approx 0.14$
$P(\text{Snowy}) = \frac{3}{7} \approx 0.43$

For features Temperature (Hot, Cold) and Humidity (High, Normal), calculate likelihoods for each class.

For Sunny:

$P(\text{Hot | Sunny}) = \frac{2}{3} \approx 0.67$ , $P(\text{Cold | Sunny}) = \frac{1}{3} \approx 0.33$
$P(\text{High | Sunny}) = \frac{2}{3} \approx 0.67$ , $P(\text{Normal | Sunny}) = \frac{1}{3} \approx 0.33$

For Rainy:

$P(\text{Hot | Rainy}) = 1.00$ , $P(\text{Cold | Rainy}) = 0.00$
$P(\text{High | Rainy}) = 0.00$ , $P(\text{Normal | Rainy}) = 1.00$

For Snowy:

$P(\text{Hot | Snowy}) = 0.00$ , $P(\text{Cold | Snowy}) = 1.00$
$P(\text{High | Snowy}) = \frac{1}{3} \approx 0.33$ , $P(\text{Normal | Snowy}) = \frac{2}{3} \approx 0.67$

These calculations illustrate how to derive prior probabilities and likelihoods for Naive Bayes Classification.