Welcome to the world of Linear Discriminant Analysis (LDA), a technique widely used for dimensionality reduction in supervised learning. In this lesson, we're delving into LDA and building a Python implementation. We'll use the popular Iris dataset for a practical example.

LDA reduces dimensionality by constructing a feature space that optimally separates the classes in the data. The axes in this space are linear combinations of the original features and are known as eigenvectors. The LDA algorithm consists of several steps, such as calculating class mean vectors and scatter matrices, then finding eigenvalues and eigenvectors that map the original feature space on to a lower-dimensional space.

To understand LDA, let's start with a simple two-dimensional example. Suppose we have data points scattered across a 2-D space, with each point belonging to one of two possible classes.

In LDA, we want to project these points onto a line such that when the points are projected, the two classes are as separated as possible. The goal here is twofold:

1. Maximize the distance between the means of the two classes.
2. Minimize the variation (in other words, the scatter) within each category.

This forms the intuition behind LDA. The crux of an LDA transformation involves formulating a weight matrix and transforming our input data by multiplying it with this weight matrix. The weights here help in increasing class separability. Let's see how we can achieve this mathematically using scatter matrices.

Let's load the Iris dataset for LDA which is a 3-class dataset with 4 features (sepal length, sepal width, petal length, petal width)

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris

# Load the Iris dataset
iris = load_iris()
X = iris.data
y = iris.target

# Scale the data to zero mean and unit variance
X = (X - np.mean(X, axis=0)) / np.std(X, axis=0)

The code above loads the Iris dataset and normalizes the data. Normalization helps balance different input variables before they're input into the machine learning algorithm.

To build our LDA from scratch, we start by defining a Python class called LDA. The class has two methods for calculating within-class and between-class scatter matrices, and finally, a fit method for calculating the transformation matrix.

class LDA:
    def __init__(self):
        self.W = None

The LDA class has self as its first parameter so that instance variables and methods can be accessed in the class. We define the __init__ method that Python calls when you create a new instance of this class. This method sets up a state for the object by defining the variable W which will hold the transformation matrix.

Next, we create an instance of the LDA class and call the fit method passing in our original feature space, the target variable, and the number of components we need for transformation.

We then perform projection of the dataset from the original features space to the new subspace. Afterward, we plot the projected samples.

lda = LDA()
lda.fit(X, y, n_components=2)
X_lda = lda.transform(X)

# Scatter plot of transformed data
colors = ['red', 'blue', 'green']
labels = iris.target_names
for label, color in zip(np.unique(y), colors):
    plt.scatter(X_lda[y == label, 0], X_lda[y == label, 1], color=color, label=labels[label])

plt.title('LDA: Projected data onto the first two components')
plt.xlabel('LD1')
plt.ylabel('LD2')
plt.legend()
plt.show()

Finally, we can plot the transformed data:

Great job! You've successfully understood the concepts of LDA and its mathematical foundation, and built a simple LDA from scratch using Python. Now, get ready for practice exercises to apply your newly acquired knowledge and consolidate learning! Happy learning!