Deep Learning · TEE · section

What an optimizer is trying to do

Optimization algorithms

Definition. Optimization is the process of selecting model parameters to minimize a scalar objective. In neural-network training, a gradient-based optimizer updates weights and biases using derivatives of the training objective.

Objective and gradient

For empirical risk with an optional regularizer:

LaTeX: J(\theta)=\frac1N\sum_{i=1}^{N}L_i(\theta)+\lambda\Omega(\theta)

The loss value indicates the current objective. Its gradient indicates local sensitivity to parameters. Backpropagation computes this gradient efficiently; the optimizer determines how the parameters are updated from it. A gradient and a loss value cannot be interchanged in the update equation.

Why gradient descent subtracts the gradient

For a small displacement Δθ, first-order Taylor expansion approximates the objective by:

LaTeX: J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta

Choose Δθ = −η∇J(θ), with η > 0. Substitution gives:

LaTeX: J(\theta-\eta\nabla J)\approx J(\theta)-\eta\|\nabla J\|_2^2

The squared norm is nonnegative. Therefore the negative-gradient direction is locally a descent direction when the gradient is nonzero. The approximation explains the sign; it does not guarantee that an arbitrarily large finite step decreases loss.

Diagram: A gradient specifies a local direction; the learning rate determines the finite parameter step.. Labels: Move against the slope to reduce local loss | Move against the slope to reduce local loss | Illustration: L = w²; start at w = 2 and use learning rate 0.1. | w | old w = 2 | new w = 1.6 | Gradient = 2w = 4 | Correction = 0.1 × 4 | New w = 2 − 0.4 | A positive gradient is subtracted. A negative gradient is also subtracted, so w | increases.

Local minima, global minima and saddle points

A local minimum has objective value no larger than nearby points. A global minimum is minimal over the whole feasible domain. At a saddle point, the objective increases in some directions and decreases in others. A small gradient alone does not establish a minimum: a stationary point can also be a maximum or saddle.

Neural-network objectives are generally nonconvex. Training can encounter flat regions, ill-conditioned valleys, noisy gradient estimates and poor step sizes. Momentum modifies the update direction using previous steps; adaptive methods rescale parameter-specific updates. These methods improve particular optimization behavior, without guaranteeing the global optimum or best test performance.