Welcome back to our course "Neural Network Fundamentals: Neurons and Layers"! In this third lesson, we're building upon what you learned in our previous lesson about the basic artificial neuron. You've already implemented a simple neuron that computes a weighted sum of inputs plus a bias.
Today, we're taking an important step forward by introducing activation functions — a crucial component that enables neural networks to learn complex patterns. In particular, we'll focus on the Sigmoid activation function, one of the classical functions used in neural networks.
In our previous lesson, our neuron could only produce linear outputs. While this is useful for some tasks, it severely limits what our neural networks can learn. Today, we'll overcome this limitation by adding non-linearity to our neurons, allowing them to model more complex relationships in data.
The Need for Non-Linearity
Understanding Activation Functions
An activation function determines whether a neuron should be "activated" or not, based on the weighted sum of its inputs.
In biological terms, this mimics how neurons in our brains "fire" when stimulated sufficiently. In computational terms, activation functions introduce non-linearity into the network, allowing it to learn complex patterns.
Activation functions typically have these characteristics:
They are non-linear, allowing the network to model complex relationships
They are differentiable, which is important for the training process (we'll explore this in future lessons)
They usually map inputs to a bounded range (like [0,1] or [-1,1])
Some common activation functions include:
Sigmoid: Maps inputs to values between 0 and 1
Tanh: Maps inputs to values between -1 and 1
ReLU (Rectified Linear Unit): Returns the input if positive; otherwise, returns 0
In this lesson, we'll focus on implementing and using the Sigmoid function, which was historically one of the first activation functions used in neural networks.
The Sigmoid Activation Function
Implementing the Sigmoid Function
Enhancing Our Neuron with Activation
Now that we have our sigmoid function, let's enhance our Neuron class from the previous lesson to include this activation function. We'll modify the forward method to apply the sigmoid function to the weighted sum:
class Neuron: def __init__(self, n_inputs): """ Initialize a neuron. n_inputs: Number of input features. """ self.weights = np.random.rand(n_inputs) * 0.1 # Small random weights self.bias = 0.0 # Start with zero bias def forward(self, inputs): """ Calculate the neuron's output: 1. Weighted sum of inputs and weights, plus bias. 2. Apply the sigmoid activation function. inputs: A NumPy array of input features. """ if inputs.shape[0] != self.weights.shape[0]: raise ValueError( f"Input size {inputs.shape[0]} does not match neuron's " f"expected input size {self.weights.shape[0]}." ) weighted_sum = np.dot(inputs, self.weights) # Calculate dot product raw_output = weighted_sum + self.bias # Add bias # Apply activation function to introduce non-linearity activated_output = sigmoid(raw_output) return activated_output, raw_output # Return both for demonstration
The key changes in this updated version are:
We still calculate the weighted sum and add the bias, but now we call this intermediate result raw_output.
We apply the sigmoid function to the raw output, producing an activated_output.
We return both the activated output and the raw output, which will help us understand the effect of the activation function.
This two-step process — first calculating the linear combination and then applying the non-linear activation — is fundamental to how neurons operate in neural networks.
Testing the Activated Neuron
Now let's test our enhanced neuron to see how the activation function affects its output:
if __name__ == "__main__": # Initialize a neuron with 3 input features num_input_features = 3 neuron = Neuron(num_input_features) print(f"Neuron weights: {neuron.weights}, Bias: {neuron.bias}") # Create sample input and process it through the neuron sample_inputs = np.array([1.0, 2.0, 3.0]) print(f"\nInput to neuron: {sample_inputs}") # Get both raw and activated outputs activated_output, raw_output = neuron.forward(sample_inputs) print(f"Neuron's raw output (weighted sum + bias): {raw_output:.4f}") print(f"Neuron's activated output (Sigmoid): {activated_output:.4f}")
When we run this code, we'll observe:
The random weights and bias of our initialized neuron;
The input values we're feeding into the neuron;
The raw output (weighted sum + bias) before applying the activation function;
The final activated output after applying the sigmoid function.
Here is a sample output:
Neuron weights: [0.02619957 0.02087375 0.09072497], Bias: 0.0Input to neuron: [1. 2. 3.]Neuron's raw output (weighted sum + bias): 0.3401Neuron's activated output (Sigmoid): 0.5842
The key insight here is seeing how the sigmoid function transforms the raw output. If the raw output is a large positive number, the sigmoid output will be close to 1. If the raw output is a large negative number, the sigmoid output will be close to 0. And if the raw output is around 0, the sigmoid output will be around 0.5.
This non-linear transformation is what enables neural networks to model complex patterns and make decisions. The sigmoid function essentially "squashes" the raw output into a range between 0 and 1, which can be interpreted as a probability or a level of activation.
Conclusion and Next Steps
Excellent work! You've enhanced your neuron by adding the sigmoid activation function, introducing the critical non-linearity that neural networks need to model complex patterns. We've explored why non-linearity is essential, how the sigmoid function works mathematically, and how to implement it efficiently in code. This foundational understanding of activation functions is vital as you continue your journey into neural networks.
In the upcoming practice section, you'll gain hands-on experience working with activation functions and observe their effects on various inputs. After mastering the concepts in this lesson, we'll move forward by combining multiple neurons into layers, creating more sophisticated neural network architectures that can solve increasingly complex problems. Keep up the great work!
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Before diving into specific activation functions, let's understand why we need them in the first place.
The neuron we built in the previous lesson computes a weighted sum of inputs plus a bias. This is a linear transformation — mathematically, it can only represent straight lines (in 2D) or flat planes (in higher dimensions).
Consider what would happen if we stacked multiple layers of these linear neurons:
Layer 1 output is a linear function of the input:
Layer1out=W1X+b1
Layer 2 output is a linear function of Layer 1's output:
We can define new parameters W3=(W2W1) and b3=(W2b1+b2), so:
Layer2out=W3X+b3
As we can see, this simplifies to just another linear function! No matter how many linear layers we stack, the entire network would still only be able to learn linear relationships.
But real-world problems are rarely linear. Think about image recognition — the relationship between pixel values and whether an image contains a cat is highly non-linear. To model such complex patterns, we need to introduce non-linearity into our networks.
This is precisely the role of activation functions — they apply a non-linear transformation to the neuron's output, enabling neural networks to learn and represent more complex patterns.
The Sigmoid (also called the logistic function) maps any input value to a value between 0 and 1. It's defined by the following formula:
sigmoid(x)=1+e−x1
Where e is the base of the natural logarithm.
The Sigmoid function has several interesting properties:
It's S-shaped (hence the name "sigmoid," meaning S-shaped).
For large positive inputs, it approaches 1, while for large negative inputs, it approaches 0.
It has a smooth gradient, making it suitable for gradient-based learning algorithms.
Its output can be interpreted as a probability (since it's between 0 and 1).
Let's visualize this function with a graph:
Now, let's implement the sigmoid function in Python using NumPy. The implementation is quite straightforward:
Python
import numpy as npdef sigmoid(x): """ Sigmoid activation function. x: A scalar or NumPy array. Returns the sigmoid of x. """ return 1 / (1 + np.exp(-x))
This function takes advantage of NumPy's ability to process arrays efficiently. Whether we pass a single value or an array of values, NumPy will apply the sigmoid transformation to each element.
Notice how closely our code matches the mathematical definition. The expression np.exp(-x) calculates e−x for each element in x, and then we perform the division as specified in the formula.
Let's test this function with a few values to make sure it works as expected:
Python
# Test with a few valuesprint(f"sigmoid(0) = {sigmoid(0)}") # Should be 0.5print(f"sigmoid(2) = {sigmoid(2):.4f}") # Should be around 0.88print(f"sigmoid(-2) = {sigmoid(-2):.4f}") # Should be around 0.12print(f"sigmoid(np.array([-2, 0, 2])) = {sigmoid(np.array([-2, 0, 2]))}")
When we run this code, we'll see that our sigmoid function correctly maps the input values to their corresponding sigmoid outputs, demonstrating both its behavior for individual values and its ability to process arrays efficiently.
Python
class Neuron: def __init__(self, n_inputs): """ Initialize a neuron. n_inputs: Number of input features. """ self.weights = np.random.rand(n_inputs) * 0.1 # Small random weights self.bias = 0.0 # Start with zero bias def forward(self, inputs): """ Calculate the neuron's output: 1. Weighted sum of inputs and weights, plus bias. 2. Apply the sigmoid activation function. inputs: A NumPy array of input features. """ if inputs.shape[0] != self.weights.shape[0]: raise ValueError( f"Input size {inputs.shape[0]} does not match neuron's " f"expected input size {self.weights.shape[0]}." ) weighted_sum = np.dot(inputs, self.weights) # Calculate dot product raw_output = weighted_sum + self.bias # Add bias # Apply activation function to introduce non-linearity activated_output = sigmoid(raw_output) return activated_output, raw_output # Return both for demonstration
Python
if __name__ == "__main__": # Initialize a neuron with 3 input features num_input_features = 3 neuron = Neuron(num_input_features) print(f"Neuron weights: {neuron.weights}, Bias: {neuron.bias}") # Create sample input and process it through the neuron sample_inputs = np.array([1.0, 2.0, 3.0]) print(f"\nInput to neuron: {sample_inputs}") # Get both raw and activated outputs activated_output, raw_output = neuron.forward(sample_inputs) print(f"Neuron's raw output (weighted sum + bias): {raw_output:.4f}") print(f"Neuron's activated output (Sigmoid): {activated_output:.4f}")
text
Neuron weights: [0.02619957 0.02087375 0.09072497], Bias: 0.0Input to neuron: [1. 2. 3.]Neuron's raw output (weighted sum + bias): 0.3401Neuron's activated output (Sigmoid): 0.5842
