Mastering Cluster Validation with Silhouette Scores and Visualization in Python

Introduction

Welcome! In today's lesson, we'll delve into cluster validation . We will interpret and implement the Silhouette Score, and learn how to visualize clusters for validation in Python. All of these concepts form a unified understanding that we'll explore.

Understanding Cluster Validation and Decoding the Silhouette Score

s(i)

Interpreting the Silhouette Score

Knowing how to interpret the Silhouette Score is essential. The Silhouette Score ranges between -1 and 1. The value of the Silhouette Score has the following interpretation: Score close to 1: The item is well-matched to its own cluster and poorly matched to neighboring clusters. This would be an indication of strong clustering. Score close to 0: The item is on or very close to the decision boundary between two neighboring clusters. The data point is right at the boundary of the clusters. It's not distinctly in one cluster or another. Here, our clustering model is uncertain about the assignment of these points. Score close to -1: The item is mismatched to its own cluster and matched to a neighboring cluster. This case indicates that we've likely assigned a point to the wrong cluster, as it is closer to the neighboring cluster than its own. It would be ideal that all objects had a Silhouette Score of 1, but in practice, it’s almost impossible.

Python Implementation of the Silhouette Score and Visualization of Clusters for Validation

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

import numpy as np

def euclidean_distance(a, b):
    # Calculate the Euclidean distance between points a and b.
    return np.sqrt(np.sum((np.array(a) - np.array(b)) ** 2))

Python

import numpy as np

def calculate_a(point, cluster):
    # Calculate the average distance from 'point' to other points in the same cluster.
    if len(cluster) <= 1:
        return 0
    distances = [euclidean_distance(point, other) for other in cluster if not np.array_equal(point, other)]
    return sum(distances) / (len(cluster) - 1)

Python

def calculate_b(point, clusters):
    # Calculate the lowest average distance from 'point' to points in other clusters.
    min_average_distance = float('inf')
    for cluster in clusters:
        # Check if point is in the current cluster by comparing all elements
        if any(np.array_equal(point, other) for other in cluster):
            continue
        distances = [euclidean_distance(point, other) for other in cluster]
        average_distance = sum(distances) / len(cluster)
        if average_distance < min_average_distance:
            min_average_distance = average_distance
    return min_average_distance

Python

from collections import defaultdict

def custom_silhouette_score(points, labels):
    # Group points by cluster label.
    clusters = defaultdict(list)
    for point, label in zip(points, labels):
        clusters[label].append(point)

    # Convert clusters to a list for easier access.
    cluster_list = list(clusters.values())

    # Calculate silhouette score for each point.
    scores = []
    for point, label in zip(points, labels):
        a = calculate_a(point, clusters[label])
        b = calculate_b(point, cluster_list)
        score = (b - a) / max(a, b) if max(a, b) > 0 else 0
        scores.append(score)

    # Return the average silhouette score.
    return sum(scores) / len(scores)

Python

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
from sklearn import datasets

X = datasets.load_iris().data

# Fit the KMeans model
kmeans_model = KMeans(n_clusters=3, random_state=0, n_init=10).fit(X)

# Plot the clusters
plt.scatter(X[:, 0], X[:, 1], c=kmeans_model.labels_)
plt.show()

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
from sklearn import datasets

X = datasets.load_iris().data

# Fit the KMeans model
kmeans_model = KMeans(n_clusters=3, random_state=0, n_init=10).fit(X)

# Plot the clusters
plt.scatter(X[:, 0], X[:, 1], c=kmeans_model.labels_)
plt.show()

Python

# Calculate and print the average silhouette score.
average_score = custom_silhouette_score(X, kmeans_model.labels_)
print(f"Silhouette Score (Custom): {average_score}") # ~0.55

# Calculate and print the average silhouette score.
average_score = custom_silhouette_score(X, kmeans_model.labels_)
print(f"Silhouette Score (Custom): {average_score}") # ~0.55

Python

from sklearn.metrics import silhouette_score

# Calculate Silhouette Score using sklearn
score = silhouette_score(X, kmeans_model.labels_, metric='euclidean')

# Print the score
print("Silhouette score (sklearn): ", score) # ~0.55

from sklearn.metrics import silhouette_score

# Calculate Silhouette Score using sklearn
score = silhouette_score(X, kmeans_model.labels_, metric='euclidean')

# Print the score
print("Silhouette score (sklearn): ", score) # ~0.55

Python

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
from sklearn import datasets

X = datasets.load_iris().data

# Fit the KMeans model
kmeans_model = KMeans(n_clusters=3, random_state=0, n_init=10).fit(X)

# Plot the clusters
plt.scatter(X[:, 0], X[:, 1], c=kmeans_model.labels_)
plt.show()

Python

# Calculate and print the average silhouette score.
average_score = custom_silhouette_score(X, kmeans_model.labels_)
print(f"Silhouette Score (Custom): {average_score}") # ~0.55

Python

from sklearn.metrics import silhouette_score

# Calculate Silhouette Score using sklearn
score = silhouette_score(X, kmeans_model.labels_, metric='euclidean')

# Print the score
print("Silhouette score (sklearn): ", score) # ~0.55

Introduction

Understanding Cluster Validation and Decoding the Silhouette Score

Interpreting the Silhouette Score

Python Implementation of the Silhouette Score and Visualization of Clusters for Validation

Join the 1M+ learners on CodeSignal

Practical Examples

Calculating the Silhouette Score with the Custom Implementation

Silhouette Score Calculation Using sklearn

Lesson Summary and Practice