Understanding Dendrograms with Python in Hierarchical Clustering

Welcome! Following our exploration of Hierarchical Clustering, our journey today takes us to dendrograms. We regard these visual tools as par excellence because dendrograms illuminate hierarchical clustering in a form that's pleasing to the eye. We will learn to read, analyze, and interpret dendrograms using Python.

Before moving forward, let's familiarize ourselves with another common way to implement hierarchical clustering in Python - using the Scipy library's function linkage(). The linkage() function does agglomerative hierarchical clustering. It takes an array of data points and the clustering method as its primary inputs. In our case, the clustering method will be 'ward', which aims to minimize the variance within each cluster.

Here is how we can perform Hierarchical Agglomerative clustering on our cities dataset:

import numpy as np
from scipy.cluster.hierarchy import linkage

# Data provided
data = {
    'city': ['Mexico City', 'Los Angeles', 'Bogota', 'Lima', 'Madrid', 'Berlin', 'Kinshasa', 'Lagos', 'New Delhi', 'Beijing', 'Sydney'],
    'long': [-99.1332, -118.2437, -74.0721, -77.0428, -3.7038, 13.4050, 15.2663, 3.3792, 77.2090, 116.4074, 151.2093],
    'lat': [19.4326, 34.0522, 4.7110, -12.0464, 40.4168, 52.5200, -4.4419, 6.5244, 28.6139, 39.9042, -33.8688]
}

# Convert data to an array of coordinates
coordinates = np.column_stack((data['long'], data['lat']))

# Perform hierarchical clustering
Z = linkage(coordinates, method='ward')

You may wonder how linkage() compares with AgglomerativeClustering from the sklearn library. While the underlying concept is the same, there are some differences primarily in their use cases.

linkage() shines for smaller datasets when you want rapid access to the full dendrogram or specify your custom distance functions. On the other hand, AgglomerativeClustering is useful for large datasets due to its memory efficiency and flexibility to return a specified number of clusters.

Now that we've computed the hierarchical clustering, it's time to visualize the results. Scipy also provides us with a useful function for this purpose named dendrogram().

from scipy.cluster.hierarchy import dendrogram
import matplotlib.pyplot as plt

# Plotting the dendrogram
plt.figure(figsize=(10, 8))
dendrogram(Z, labels=data['city'], leaf_rotation=90, leaf_font_size=12)
plt.title('Dendrogram of Cities based on Geographic Coordinates')
plt.xlabel('Cities')
plt.ylabel('Distance')
plt.show()

The result looks as follows:

Now, let's interpret the dendrogram. The dendrogram shows how the cities are clustered based on their geographic coordinates. The height of the dendrogram shows the distance between the clusters. The longer the vertical line, the further apart the clusters are. The horizontal lines show the merging of clusters. The height at which the horizontal line is cut by the vertical line indicates the distance at which the clusters were merged. The dendrogram can help identify which cities are closer to each other geographically based on the clustering. For example, Mexico City and Los Angeles are clustered together, indicating that they are closer to each other geographically compared to other cities in the dataset. Similarly, Madrid and Berlin are clustered together, indicating that they are closer to each other geographically as well.

Roughly put, the algorithm first clusters the cities in the same country, then clusters the cities in the same continent, and finally clusters all the cities together — hence the dendrogram's structure.