Exploring the Derivative as a Function

Understanding the derivative as a function is essential in calculus and machine learning. It helps us see how inputs affect outputs, which is crucial for optimization and learning algorithms. In this lesson, you'll learn what it means for a function to be differentiable, see examples of functions that aren't differentiable at certain points, and plot both a function and its derivative to explore their relationships. By the end, you'll identify where a function grows, decays, and has extremum points through its derivative.

A function is differentiable at a point if it has a defined derivative there. This means the function should be smooth without sharp turns or corners at that point. For instance, the function $f(x) = |x|$ is not differentiable at $x = 0$ because it has a sharp corner there, making it impossible to define a single tangent line.

To visualize this:

The derivative itself can be seen as a function that varies with $x$. For example, for the function $f(x) = x^2$, we can define the derivative as $f'(x) = 2x$, which is also a function of $x$.

There are rules to obtain this formula. However, we don't cover them in this lesson, as for machine learning its much more important to understand the idea of the derivative rather than learning to calculate it.

Let's visualize this relationship:

From the plot:

1. Derivative is Negative When the Function Decays: For $x < 0$, $f(x) = x^2$ is decreasing, and $f'(x) = 2x$ is negative, indicating a downward slope.

2. Derivative is Zero at a Extremum Point: At $x = 0$, $f(x) = x^2$ has a minimum point, and $f'(x) = 2x$ is zero, showing a extremum point. An extremum point is either a local minimum or a local maximum of the function. Note that there are other cases where $f'(x)$ could be zero, for example it could happen at the inflection point, but we won't consider it in this course.

3. Derivative is Positive When the Function Grows: For $x > 0$, $f(x) = x^2$ is increasing, and $f'(x) = 2x$ is positive, indicating an upward slope.

Note that we are talking about a local minimum/maximum here. For example, take a look at this graph:

Here, $x=0$ is the point of local minimum. The function grows both to the right and to the left of it. However, it is not a global minimum: we can clearly observe the function getting lower to the left.