Basic Derivatives

Welcome to our lesson on basic derivatives! In machine learning, understanding how things change is crucial. Derivatives help measure these changes, which is key for optimizing models. By the end, you'll understand derivatives, know how to calculate them, and see a practical example using Python.

Let's understand the concept of a derivative. Imagine you're driving a car. Your speed indicates how your position changes over time. Similarly, a derivative measures how a function's output changes as its input changes. It's like the speed for functions. The derivative is calculated for every input of the original function. So, the derivative is also a function of the same input.

Let's try to approximate the derivative of the function $f(x) = x^2$ at $x=2$. In other words, we want to see the speed of the function's growth (or decay) at this point. Using a small change $\Delta x$, we will find how much function changed when the x changed:

Here’s a graph to illustrate this:

In the picture, the green dashed line shows our slope approximation using $\Delta x = 1$. We see, that the function changed from 4 to 9, while x changed from 2 to 3. We can make a conclusion that the speed of the function at $x=2$ is roughly $\frac{9-4}{1}$, which equal to 5.

We care about the derivative at point $x=2$, but what we really calculate is the speed of the function on the $[2, 3]$ interval. Let's improve the accuracy of our approximation by choosing a smaller $\Delta x$. Here is a new graph:

Here, the $\Delta x=0.5$. The derivative approximation is $\frac{f(2.5) - f(2)}{0.5}$, which is equal to 4.5. This time, we calculated the function's growth speed on the $[2, 2.5]$ interval, which gives us a better approximation quality. By shrinking this interval (in other words, by making $\Delta x$ lower), we can improve the approximation quality.

For example, making the same calculations on the $[2, 2.00001]$ interval will give you a value, which is very close to the actual derivative at point $x=2$. However, it won't be exact.