Mastering K-means Clustering: Selection of Clusters and Centroid Initialization

Introduction

Greetings! Our journey into K-means clustering deepens as we explore two crucial elements: the selection of the number of clusters and the initialization of centroids. Our aim is to comprehend these aspects and put them into action using Python. Let's move forward!

Choosing Clusters and Initializing Centroids in K-means

The K in K-means signifies the number of clusters. Centroids, the centers of each cluster, are equally significant. Their initial placement in K-means is crucial. Poorly initialized centroids can lead to sub-optimal clustering — a reason why multiple runs with different initial placements are essential. This highlights the importance of choosing both the number of clusters and their initial centroids.

Revising K-means Algorithm

Sklearn's KMeans not only allows us to specify the number of clusters and maximum iterations but also provides an important parameter, init, where we can set the initial centroids to be used. Let's import the KMeans class from scikit-learn library and see how we can initialize our centroids there. Python from sklearn.cluster import KMeans import numpy as np from sklearn.datasets import load_iris import matplotlib.pyplot as plt data = load_iris().data # Initialize the number of clusters and centroids num_clusters = 3 initial_centroids = data[np.random.choice(range(data.shape[0]), num_clusters, replace=False)] kmeans = KMeans(n_clusters=num_clusters, init=initial_centroids, n_init=1) In the code above, n_clusters sets the total number of clusters, init is an optional parameter that accepts the initial centroid positions. By using n_init=1, we disable sklearn's built-in multiple runs with different centroid seeds because we want to use our manually initiated centroids. After defining the KMeans object with our specified parameters, we fit the model to our Iris dataset. kmeans.fit(data) computes K-means clustering using our data and initial centroid positions: Python kmeans.fit(data) labels = kmeans.labels_centroids = kmeans.cluster_centers_# Visualization plt.scatter(data[:, 0], data[:, 1], c=labels, cmap='viridis') plt.scatter(centroids[:, 0], centroids[:, 1], c='red') plt.show() Here, kmeans.labels_gives us the labels of each point, and kmeans.cluster_centers_provides the coordinates of cluster centers. Like before, we represent the data points in different clusters by colors, and the centroids are marked in red. See how easy it is to use sklearn's KMeans once we understand the underlying theory! Using Python libraries like this enhances our efficiency and saves time, especially when working on more complex projects.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import numpy as np
np.random.seed(42)

# We're using a simple 2D dataset for ease of comparing
data = np.array([[3, 1], [5, 1], [2, 3], 
                 [8, 2], [9, 3], [7, 1], 
                 [15, 15], [13, 16], [14, 14]])

# Initialize the number of clusters and centroids and fit KMeans model
num_clusters = 2
kmeans = KMeans(n_clusters=num_clusters, n_init=10, random_state=42)
kmeans.fit(data)

# Retrieve labels and centroids
labels2 = kmeans.labels_
centroids2 = kmeans.cluster_centers_
print(labels2) # Prints [0 0 0 0 0 0 1 1 1], Might be different depending on the versions of libraries

Python

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import numpy as np
np.random.seed(42)

# We're using a simple 2D dataset for ease of comparing
data = np.array([[3, 1], [5, 1], [2, 3], 
                 [8, 2], [9, 3], [7, 1], 
                 [15, 15], [13, 16], [14, 14]])

# Initialize the number of clusters and centroids and fit KMeans model
num_clusters = 3
kmeans = KMeans(n_clusters=num_clusters, n_init=10, random_state=42)
kmeans.fit(data)

# Retrieve labels and centroids
labels3 = kmeans.labels_
centroids3 = kmeans.cluster_centers_
print(labels3) # Prints [2 2 2 0 0 0 1 1 1], Might be different depending on the versions of libraries

Python

# Plot both k=2 and k=3 on the same plot using subplots
fig, axs = plt.subplots(1, 2, figsize=(10, 5))

axs[0].scatter(data[:, 0], data[:, 1], c=labels2, cmap='viridis')
axs[0].scatter(centroids2[:, 0], centroids2[:, 1], c='red')
axs[0].set_title("K-means Clustering with k = 2")

axs[1].scatter(data[:, 0], data[:, 1], c=labels3, cmap='viridis')
axs[1].scatter(centroids3[:, 0], centroids3[:, 1], c='red')
axs[1].set_title("K-means Clustering with k = 3")

plt.show()

Python

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

np.random.seed(0)

# Data preparation
x1 = np.random.normal(loc=5, scale=1, size=(100, 2))
x2 = np.random.normal(loc=10, scale=2, size=(100, 2))
x = np.concatenate([x1, x2])

Python

# K-means clustering preparation
k = 3

def calc_distance(x1, x2):
    return (sum((x1 - x2)**2))**0.5

def find_closest_centroids(centroids, x):
    assigned_centroid = []
    for i in x:
        distance = []
        for j in centroids:
            distance.append(calc_distance(i, j))
        assigned_centroid.append(np.argmin(distance))
    return assigned_centroid

def calc_centroids(clusters, x):
    new_centroids = []
    new_df = pd.concat([pd.DataFrame(x), pd.DataFrame(clusters, columns=['cluster'])], axis=1)
    for c in set(new_df['cluster']):
        current_cluster = new_df[new_df['cluster'] == c][new_df.columns[:-1]]
        cluster_mean = current_cluster.mean(axis=0)
        new_centroids.append(cluster_mean)
    return new_centroids

# K-means clustering function
def kmeans_clustering(x, initial_centroids):
    centroids = initial_centroids
    
    for i in range(10):
        get_centroids = find_closest_centroids(centroids, x)
        centroids = calc_centroids(get_centroids, x)

    return centroids, get_centroids

Python

# Initial centroids for two different initializations
np.random.seed(42)  # Keeping the seed constant for reproducibility
initial_centroids1 = x[np.random.choice(range(x.shape[0]), size=k, replace=False), :]
initial_centroids2 = x[np.random.choice(range(x.shape[0]), size=k, replace=False), :]

# Applying K-means clustering with different initial centroids will affect the final clustering
centroids1, get_centroids1 = kmeans_clustering(x, initial_centroids1)
centroids2, get_centroids2 = kmeans_clustering(x, initial_centroids2)

# Visualization of the final clustering for both sets of initial centroids
fig, axs = plt.subplots(1, 2, figsize=(12, 6))

axs[0].scatter(x[:,0], x[:,1], c=get_centroids1)
axs[0].scatter(np.array(centroids1)[:,0], np.array(centroids1)[:,1], c='red', marker='X')
axs[0].set_title("First Initialization")

axs[1].scatter(x[:,0], x[:,1], c=get_centroids2)
axs[1].scatter(np.array(centroids2)[:,0], np.array(centroids2)[:,1], c='red', marker='X')
axs[1].set_title("Second Initialization")

plt.show()

Introduction

Choosing Clusters and Initializing Centroids in K-means

Revising K-means Algorithm

Join the 1M+ learners on CodeSignal

Understand the Implications

Selection of the Number of Clusters

Initial Centroid Initialization: Potential Pitfalls and Solutions

Lesson Summary and Practice