Activation Functions in C++

Welcome to our exploration of "Understanding Activation Functions". In this lesson, we will investigate activation functions — essential components in neural networks that determine the output of a neuron. We will focus on the theory and C++ implementations of five activation functions. Let's begin our journey into the world of neural networks using C++.

Activation functions play a crucial role in neural networks by determining the output of each neuron. You can think of them as gates: they decide whether a neuron should be activated or not, based on the input. In this lesson, we will explore five types of activation functions.

Mathematical Formula:

Let's start with the step function, also known as the threshold function. This simple activation function acts like a switch: if the input value is greater than or equal to a threshold (commonly 0), the function returns 1; otherwise, it returns 0.

C++ Implementation:

int step_function(double x) {
    return (x >= 0) ? 1 : 0;
}

Step Function Visualization

To visualize the step function in C++, you can use the matplotlibcpp library, which provides plotting capabilities similar to those in other languages. Here is an example of how you might plot the step function:

#include <vector>
#include "matplotlibcpp.h"

namespace plt = matplotlibcpp;

int step_function(double x) {
    return (x >= 0) ? 1 : 0;
}

int main() {
    std::vector<double> x, y;
    for (double i = -10; i <= 10; i += 0.2) {
        x.push_back(i);
        y.push_back(step_function(i));
    }
    plt::plot(x, y);
    plt::save("static/images/step_function.png");
    return 0;
}

This code generates and saves a plot of the step function:

Mathematical Formula:

The sigmoid function maps any real value to a value between 0 and 1, producing an S-shaped curve. It is often used when the output needs to represent a probability.

C++ Implementation:

#include <cmath>

double sigmoid_function(double x) {
    return 1.0 / (1.0 + std::exp(-x));
}

Sigmoid Function Visualization

You can visualize the Sigmoid function in C++ using matplotlibcpp as before:

Mathematical Formula:

The ReLU (Rectified Linear Unit) function returns the input value itself if it is positive; otherwise, it returns zero. It is widely used in neural networks due to its simplicity and effectiveness.

C++ Implementation:

double relu_function(double x) {
    return (x > 0) ? x : 0;
}

ReLU Function Visualization

You can visualize the ReLU function in C++ using matplotlibcpp as before:

Mathematical Formula:

The tanh (hyperbolic tangent) function is similar to the sigmoid function but maps input values to a range between -1 and 1. It is useful when you want the output to represent both positive and negative values.

C++ Implementation:

#include <cmath>

double tanh_function(double x) {
    return (2.0 / (1.0 + std::exp(-2.0 * x))) - 1.0;
}

Tanh Function Visualization

You can visualize the tanh function in C++ using matplotlibcpp as before:

Mathematical Formula:

The softplus function is a smooth approximation of the ReLU function and is differentiable everywhere. It is defined as the natural logarithm of (1 + exp(x)).

C++ Implementation:

#include <cmath>

double softplus_function(double x) {
    return std::log(1.0 + std::exp(x));
}

Softplus Function Visualization

You can visualize the Softplus function in C++ using matplotlibcpp as before:

In this lesson, we explored the theory and C++ implementations of several important activation functions: step, sigmoid, ReLU, tanh, and softplus. You have learned how to implement these functions, their mathematical formulas, and how to visualize them using C++ and the matplotlibcpp library.

By mastering these activation functions, you are building a strong foundation for understanding and constructing neural networks in C++.