Calculus for AI: Understanding Derivatives, Gradients, and Optimization in Machine Learning

1: Calculus – The Engine of AI

1. Derivatives and Gradients: Measuring Change

What are Derivatives?

At its core, a derivative measures the rate of change of a function at a given point. Imagine you’re driving a car, and the derivative is the speedometer—it tells you how fast you’re going at any given moment.

Mathematical Definition

If \( f(x) \) is a function, the derivative of \( f(x) \) with respect to \( x \) is defined as:

This measures how \( f(x) \) changes as \( x \) changes.

Interpretation

1. Positive Derivative: \( f(x) \) increases as \( x \) increases.
2. Negative Derivative: \( f(x) \) decreases as \( x \) increases.
3. Zero Derivative: \( f(x) \) is flat (critical point).

Gradient: The Multivariable Generalization

In AI, functions often depend on multiple variables (e.g., weights in a neural network). The gradient extends derivatives to multiple dimensions.

If \( f(x, y) \) is a function of two variables, the gradient is:

The gradient points in the direction of the steepest increase of the function. In optimization, we use its negative direction to minimize functions.

Numerical Example

Let’s compute the derivative of \( f(x) = x^2 \) at \( x = 3 \):

This means the slope of the function at \( x = 3 \) is 6.

Code Example: Derivatives and Gradients

Using Python with SymPy for symbolic differentiation:

import sympy as sp

# Define a variable and a function
x = sp.Symbol('x')
f = x**2

# Compute derivative
f_prime = sp.diff(f, x)
print("Derivative of f(x):", f_prime)

# Evaluate at x = 3
value_at_3 = f_prime.subs(x, 3)
print("Value of derivative at x=3:", value_at_3)

# Gradient for a multivariable function
y = sp.Symbol('y')
g = x**2 + y**2
gradient = [sp.diff(g, var) for var in (x, y)]
print("Gradient of g(x, y):", gradient)

Mermaid.js Diagram: Derivative Flow

graph LR
    Function["Function f(x)"] --> Derivative["Derivative f'(x)"]
    Derivative --> Positive["Positive Slope (Increasing)"]
    Derivative --> Negative[Negative Slope (Decreasing)"]
    Derivative --> Zero["Zero Slope (Critical Point)"]

2. Partial Derivatives: Zooming in on Multivariable Functions

What Are Partial Derivatives?

A partial derivative measures the rate of change of a multivariable function with respect to one variable, holding all other variables constant.

Mathematical Definition

For a function \( f(x, y) \), the partial derivative with respect to \( x \) is:

Similarly, the partial derivative with respect to \( y \) is:

Notation

Partial derivatives are denoted by:

• \( \frac{\partial}{\partial x} f(x, y) \)
• \( f_x(x, y) \)

Why Partial Derivatives Matter in AI

In AI, especially in training neural networks, we deal with loss functions that depend on multiple variables (e.g., weights and biases). Partial derivatives allow us to compute the rate of change of the loss with respect to each parameter, enabling optimization techniques like gradient descent.

Numerical Example

Consider the function \( f(x, y) = x^2 + 3xy + y^2 \).

1. Partial Derivative with Respect to \( x \):

   \[ \frac{\partial f}{\partial x} = 2x + 3y \]

2. Partial Derivative with Respect to \( y \):

   \[ \frac{\partial f}{\partial y} = 3x + 2y \]

If \( x = 1 \) and \( y = 2 \):

Code Example: Computing Partial Derivatives

Let’s calculate partial derivatives using SymPy in Python:

import sympy as sp

# Define variables and function
x, y = sp.symbols('x y')
f = x**2 + 3*x*y + y**2

# Compute partial derivatives
partial_x = sp.diff(f, x)
partial_y = sp.diff(f, y)

# Print results
print("Partial derivative with respect to x:", partial_x)
print("Partial derivative with respect to y:", partial_y)

# Evaluate at a point (x=1, y=2)
value_x = partial_x.subs({x: 1, y: 2})
value_y = partial_y.subs({x: 1, y: 2})
print("Value of partial derivative w.r.t x at (1, 2):", value_x)
print("Value of partial derivative w.r.t y at (1, 2):", value_y)

Geometric Interpretation

Partial derivatives indicate the slope of the function in the direction of a particular variable. Imagine standing on a hill and moving only in the \( x \)-direction. The steepness you feel is \( \frac{\partial f}{\partial x} \). Move in the \( y \)-direction, and it’s \( \frac{\partial f}{\partial y} \).

Mermaid.js Diagram: Partial Derivative Flow

graph TD
    MultivariableFunction["Multivariable Function f(x, y)"] --> PartialX[Partial Derivative w.r.t x]
    MultivariableFunction --> PartialY[Partial Derivative w.r.t y]
    PartialX --> GradientStepX[Change in x Direction]
    PartialY --> GradientStepY[Change in y Direction]

3. Chain Rule: The Backbone of Backpropagation

What is the Chain Rule?

The Chain Rule allows us to compute the derivative of a composite function—functions that are “nested” inside each other. If a function \( y \) depends on \( u \), and \( u \) depends on \( x \), then:

Think of it as a conveyor belt: you calculate the rate of change for each step and multiply them together to get the total rate of change.

Mathematical Definition

For a composite function \( f(g(x)) \), where \( u = g(x) \), the derivative is:

Generalized Chain Rule

If \( z = f(u, v) \), \( u = g(x, y) \), and \( v = h(x, y) \), the total derivative of \( z \) with respect to \( x \) is:

Why the Chain Rule is Important in AI

In neural networks, each layer applies a function to the data and passes the result to the next layer. During backpropagation, the Chain Rule is used to compute how the loss function changes with respect to every parameter (e.g., weights and biases). This allows the network to update its parameters to minimize the loss.

Numerical Example: Chain Rule

Let’s compute the derivative of \( f(x) = (3x + 2)^2 \).

1. Let \( u = 3x + 2 \), so \( f(x) = u^2 \).
2. Compute \( \frac{df}{du} \) and \( \frac{du}{dx} \): \[ \frac{df}{du} = 2u, \quad \frac{du}{dx} = 3 \]
3. Apply the Chain Rule: \[ \frac{df}{dx} = \frac{df}{du} \cdot \frac{du}{dx} = 2u \cdot 3 = 6u \]
4. Substitute \( u = 3x + 2 \): \[ \frac{df}{dx} = 6(3x + 2) \]

For \( x = 1 \):

Code Example: Chain Rule in Python

Using SymPy to automate the chain rule:

import sympy as sp

# Define variables and functions
x = sp.Symbol('x')
u = 3*x + 2
f = u**2

# Compute derivatives
df_du = sp.diff(f, u)  # Derivative of f w.r.t u
du_dx = sp.diff(u, x)  # Derivative of u w.r.t x
df_dx = sp.diff(f, x)  # Derivative of f w.r.t x using chain rule

# Print results
print("df/du:", df_du)
print("du/dx:", du_dx)
print("df/dx:", df_dx)

# Evaluate at x = 1
value_at_1 = df_dx.subs(x, 1)
print("Value of df/dx at x=1:", value_at_1)

Geometric Interpretation

The Chain Rule lets us “follow the path” of changes. Imagine climbing a mountain trail:

• The trail steepness changes based on your direction (\( du/dx \)).
• The slope at your location changes based on the altitude map (\( df/du \)).

The total difficulty of climbing the mountain combines these two factors!

Mermaid.js Diagram: Chain Rule Flow

graph TD
    X[Input Variable x] --> U[Intermediate Variable u]
    U --> F[Final Output f(u)]
    X --> DX[Change in x (du/dx)]
    U --> DU[Change in u (df/du)]
    DX --> TotalDerivative[df/dx = df/du * du/dx]
    DU --> TotalDerivative

4. Gradient Descent and Optimization: The Heartbeat of AI Training

What is Gradient Descent?

Gradient Descent is an iterative optimization algorithm used to minimize a function. The idea is simple: adjust the parameters of the function in the direction of the steepest descent (negative gradient) until we reach a minimum.

Mathematical Formulation

Given a function \( f(x) \), the update rule for gradient descent is:

Where:

• \( \eta \): Learning rate (controls step size)
• \( \nabla f(x) \): Gradient of \( f \) at \( x \)

Steps in Gradient Descent

1. Initialize Parameters: Start with a random guess for the parameters.

2. Compute the Gradient: Use derivatives (or partial derivatives for multivariable functions) to calculate the gradient at the current position.

3. Update Parameters: Adjust the parameters in the opposite direction of the gradient.

4. Repeat Until Convergence: Stop when the gradient is close to zero, indicating a minimum.

Numerical Example

Let’s minimize the function \( f(x) = x^2 \) using gradient descent.

1. Gradient:

   \[ \frac{df}{dx} = 2x \]

2. Update Rule:

   \[ x_{\text{new}} = x_{\text{old}} - \eta \cdot 2x_{\text{old}} \]

3. Iterations:

   • Start with \( x_{\text{old}} = 4 \), \( \eta = 0.1 \): \[ x_{\text{new}} = 4 - 0.1 \cdot 2(4) = 4 - 0.8 = 3.2 \]
   • Repeat: \[ x_{\text{new}} = 3.2 - 0.1 \cdot 2(3.2) = 3.2 - 0.64 = 2.56 \]

   After a few iterations, \( x \) converges to 0, the minimum of \( f(x) \).

Code Example: Gradient Descent in Python

# Define function and gradient
def f(x):
    return x**2

def gradient(x):
    return 2 * x

# Gradient Descent Algorithm
def gradient_descent(starting_point, learning_rate, iterations):
    x = starting_point
    for i in range(iterations):
        grad = gradient(x)
        x = x - learning_rate * grad
        print(f"Iteration {i+1}: x = {x}, f(x) = {f(x)}")
    return x

# Run gradient descent
starting_point = 4
learning_rate = 0.1
iterations = 10
gradient_descent(starting_point, learning_rate, iterations)

Challenges in Gradient Descent

1. Choosing the Learning Rate:

   • Too small: Slow convergence.
   • Too large: Overshooting the minimum.

2. Local Minima:

   • Gradient descent may get stuck in local minima for non-convex functions.

3. Vanishing Gradients:

   • In deep neural networks, gradients may become very small, slowing training.

Types of Gradient Descent

1. Batch Gradient Descent:

   • Computes the gradient using the entire dataset.
   • Accurate but computationally expensive for large datasets.

2. Stochastic Gradient Descent (SGD):

   • Uses a single data point to compute the gradient at each step.
   • Faster but noisier.

3. Mini-Batch Gradient Descent:

   • A hybrid approach that uses a small batch of data points.
   • Balances speed and stability.

Mermaid.js Diagram: Gradient Descent Flow

graph TD
    Initialize[Initialize Parameters] --> ComputeGradient[Compute Gradient]
    ComputeGradient --> Update[Update Parameters]
    Update --> Converge[Check for Convergence]
    Converge --> Stop[Stop if Converged]
    Converge --> ComputeGradient[Repeat if Not Converged]