Support Vector Machine (SVM) Basics

Lesson Introduction

Welcome to our lesson on Support Vector Machines (SVM)! Today, we will learn about a powerful tool used for classifying data in machine learning. Have you ever thought about how a computer could tell the difference between pictures of cats and dogs? SVM is one way to make that possible by drawing a line, or more accurately, a hyperplane, to separate different categories. By the end of this lesson, you’ll be able to load a dataset, split it into training and testing sets, and train an SVM model using Python and Scikit-Learn . Let’s get started!

Introduction to SVM

First, let’s talk about what an SVM is. Imagine you have a bag full of apples and oranges on a table. You want to separate them into two groups using a straight line. This line that separates the two groups in such a way is called a hyperplane . SVM is a classification algorithm that finds the best hyperplane that separates different classes in the data. But wait, what if the data can’t be separated by a straight line? That’s when SVM can use something called a kernel trick to transform the data into a higher dimension where a hyperplane can be used. The data points that are closest to the hyperplane are called support vectors because they “support” the hyperplane. Essentially, SVM seeks to maximize the margin between the classes through the hyperplane, leading to better generalization in classification. Here is an example image: Dots of different classes are separated using the hyperplane (which is just a line in this 2d case), which we call the "decision boundary". The closest samples form the support vectors, drawn as dashed lines. They help to keep the decision boundary in the optimal equally-distanced position. In Python, we use the SVC class from the Scikit-Learn library to create an SVM.

Quick Reminder on Dataset Loading and Splitting

Remember, to work with any dataset, we need to load it and then split it into training and testing sets. Here’s a quick reminder using the wine dataset: Python from sklearn.datasets import load_wine from sklearn.model_selection import train_test_split # Load the wine dataset X, y = load_wine(return_X_y=True) # Split the dataset into training and testing sets (60% training, 40% testing) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Comparing with Other Models

Finally, let's compare how different models perform on this dataset. We will train and evaluate Logistic Regression, Decision Tree, Naive Bayes, and k-Nearest Neighbors (kNN) models in addition to our SVM model.

Python

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC

# Dictionary of models to compare
models = {
    "Logistic Regression": LogisticRegression(max_iter=10000),
    "Decision Tree": DecisionTreeClassifier(),
    "Naive Bayes": GaussianNB(),
    "kNN": KNeighborsClassifier(),
    "SVM": SVC(kernel='linear')
}

# Train each model and print their accuracy on the test set
for name, model in models.items():
    model.fit(X_train, y_train)
    accuracy = model.score(X_test, y_test)
    print(f"{name} Accuracy: {accuracy:.2f}")

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC

# Dictionary of models to compare
models = {
    "Logistic Regression": LogisticRegression(max_iter=10000),
    "Decision Tree": DecisionTreeClassifier(),
    "Naive Bayes": GaussianNB(),
    "kNN": KNeighborsClassifier(),
    "SVM": SVC(kernel='linear')
}

# Train each model and print their accuracy on the test set
for name, model in models.items():
    model.fit(X_train, y_train)
    accuracy = model.score(X_test, y_test)
    print(f"{name} Accuracy: {accuracy:.2f}")

The output is:

text

Logistic Regression Accuracy: 0.96
Decision Tree Accuracy: 0.94
Naive Bayes Accuracy: 1.00
kNN Accuracy: 0.69
SVM Accuracy: 0.94

Logistic Regression Accuracy: 0.96
Decision Tree Accuracy: 0.94
Naive Bayes Accuracy: 1.00
kNN Accuracy: 0.69
SVM Accuracy: 0.94

Here we can see all the models compared, and they show similar results. Remember that you can always improve a model by tuning it, which we will discuss in the last course of this course path. In this case, we might choose the Naive Bayes classifier as the best model based on its current performance or opt to tune the Decision Tree classifier for potential improvements. Choosing the right model involves experimenting with the data and comparing results.

Lesson Introduction

Introduction to SVM

Quick Reminder on Dataset Loading and Splitting

Join the 1M+ learners on CodeSignal

Training the SVM Classifier

Comparing with Other Models

Lesson Summary