Math Behind Neural Networks

Math of Neural Networks and the Universal Approximation Theorem

Neural networks are computational systems inspired by the biological neural networks that constitute our and animal brains. At their core, these networks consist of layers of nodes, or "neurons," each of which applies a simple computation to its inputs. The Universal Approximation Theorem provides the theoretical foundation for these systems, offering assurance that neural networks have the capacity to model a wide variety of functions given sufficient complexity and proper configuration.

Mathematical Representation of a Neural Network

f: \mathbb{R}^n \rightarrow \mathbb{R}^m

The Role of the Activation Function

\sigma(x) = \frac{1}{1 + e^{-x}}

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

The Universal Approximation Theorem - Simplified Explanation and Code

The Universal Approximation Theorem (UAT) is a key mathematical concept guiding the functionality of neural networks. Basically, UAT declares that a neural network with just one hidden layer - a layer between the input and output - containing a finite number of neurons (nodes where computation takes place), can nearly replicate or mimic any sort of continuous function.

Imagine the role of a hidden layer as a talented ensemble of artists. If you have a picture (a function) that you'd like them to recreate, they can do it with their collective skill set. Each artist (neuron) specializes in a different type of stroke or style, and together, they combine their talents to reproduce the image. To replicate more complex pictures (functions), you might need more artists (neurons) or an artist capable of a broader range of styles (non-linear activation function). However, as the Universal Approximation Theorem insists, they will always be able to recreate the picture to the desired level of accuracy.

Here, the artist's style is analogous to the activation function in a neural network, which is typically a non-linear function that transforms the input they receive. The Universal Approximation Theorem does come with a small caveat - it specifies that the activation function must be a non-constant, bounded, and increasing function.

To implement the concept in code and understand it better, let's explore a simple example:

Python

import numpy as np
import matplotlib.pyplot as plt

# Define a target function 
def target_function(x):
    return x * np.sin(x)

# Define the points where the function will be evaluated
x = np.linspace(0, 10, 100)

# Apply the target function 
y = target_function(x) 

# Plot the target function
plt.plot(x, y, label="Target Function: $f(x) = x*\sin(x)$")

# Let's simulate an approximation using a neural network 
n_neurons = 10
np.random.seed(42) 

# Simulate random weights and biases for each neuron
weights = np.random.rand(n_neurons)
biases = np.random.rand(n_neurons)

# Simulate neurons
neurons = np.tanh(weights * x.reshape(-1, 1) + biases)

# Learn the weighting of the neurons
coefficients = np.linalg.lstsq(neurons, y, rcond=None)[0]

# approximate function
y_approx = neurons @ coefficients

plt.plot(x, y_approx, label="Neural Network Approximation")
plt.legend()
plt.show()

Thus, with just 10 neurons and the tanh activation function, you can see that our network does a decent job approximating the target function $f(x) = x*\sin(x)$ . Of course, more complex functions may require more hidden neurons or additional layers. However, according to the Universal Approximation Theorem, they can still be approximated by a neural network! Here is a visualization of the simulated network architecture.

Deep Neural Networks & The Universal Approximation Theorem

The Universal Approximation Theorem (UAT) in its original form, pertains to neural networks with just a single hidden layer. However, in practice, we often encounter many more layers, which constitutes what we call Deep Neural Networks.

In the world of Deep Learning, these deep networks have proven to make a significant difference. When you add more hidden layers, what you're essentially doing is introducing a hierarchy of concepts learned by the neural network. For example, in a deep neural network designed for image recognition, the initial layers might learn to recognize simple patterns like edges, the middle layers may combine these patterns to recognise slightly more complex shapes and the last layers might identify high-level features such as an entire object.

Interestingly, while the original UAT does not directly apply to deep networks, subsequent research and extensions of the theorem do indicate that deep networks can be more efficient at approximating complex functions compared to shallow networks. Specifically, certain functions that could be compactly represented in a deep network might require exponentially more neurons to be represented in a shallow network.

Let's revisit our previous example, using a deeper network this time:

Python

import numpy as np
import matplotlib.pyplot as plt

# Define a target function 
def target_function(x):
    return x * np.sin(x)

# Define the points where the function will be evaluated
x = np.linspace(0, 10, 100)

# Apply the target function 
y = target_function(x) 

# Plot the target function
plt.plot(x, y, label="Target Function: $f(x) = x*\sin(x)$")

# Let's simulate an approximation using a deeper neural network 
np.random.seed(42) 

# Simulate random weights and biases for each neuron in two layers
weights_1 = np.random.rand(10)
biases_1 = np.random.rand(10)
weights_2 = np.random.rand(10)
biases_2 = np.random.rand(10)

# Simulate the first layer of neurons
neurons_1 = np.tanh(weights_1 * x.reshape(-1, 1) + biases_1)

# The output of the first layer of neurons feeds into the second layer
neurons_2 = np.tanh(weights_2 * neurons_1 + biases_2)

# Learn the weighting of the neurons
coefficients = np.linalg.lstsq(neurons_2, y, rcond=None)[0]

# approximate function
y_approx = neurons_2 @ coefficients

plt.plot(x, y_approx, label="Deep Neural Network Approximation")
plt.legend()
plt.show()

With more hidden layers (each simulating a group of artists working on each detail level), our deep network could achieve high degree of accuracy while approximating a complex function. This power of deep networks to build up layers of abstraction is why they're successful in tasks like image recognition, speech recognition and natural language processing. Here is a visualization of the simulated network architecture.

import numpy as np import matplotlib.pyplot as plt # Define a target function def target_function(x): return x * np.sin(x) # Define the points where the function will be evaluated x = np.linspace(0, 10, 100) # Apply the target function y = target_function(x) # Plot the target function plt.plot(x, y, label="Target Function: $f(x) = x*\sin(x)$") # Let's simulate an approximation using a neural network n_neurons = 10 np.random.seed(42) # Simulate random weights and biases for each neuron weights = np.random.rand(n_neurons) biases = np.random.rand(n_neurons) # Simulate neurons neurons = np.tanh(weights * x.reshape(-1, 1) + biases) # Learn the weighting of the neurons coefficients = np.linalg.lstsq(neurons, y, rcond=None)[0] # approximate function y_approx = neurons @ coefficients plt.plot(x, y_approx, label="Neural Network Approximation") plt.legend() plt.show()

Math Behind Neural Networks

Math of Neural Networks and the Universal Approximation Theorem

Mathematical Representation of a Neural Network

The Role of the Activation Function

Join the 1M+ learners on CodeSignal

The Universal Approximation Theorem - Simplified Explanation and Code

Deep Neural Networks & The Universal Approximation Theorem

Summary and How TensorFlow Hides Away the Math Complexity