Deep Learning · TEE · section
Variables functions and symbols
Mathematics revision
These notes assume familiarity with algebra and basic calculus. The revision here covers the notation used in neural-network equations and the conventions that affect numerical answers.
| Notation | Technical meaning |
|---|---|
| x, y, ŷ | Input feature vector, target, model prediction |
| W, b, θ | Weight matrix, bias vector, complete trainable parameter vector |
| z, a | Preactivation (affine input) and activation (layer output) |
| L, J | Per-example loss and aggregate training objective |
| ∂L/∂w | Partial derivative with respect to one parameter |
| ∇θJ | Gradient: vector of partial derivatives with respect to θ |
| η, λ, ε | Learning rate, regularization coefficient, numerical stabilizer |
| Wᵀ, u⊙v | Transpose and elementwise multiplication |
| θₜ, W⁽ˡ⁾ | Update index t and layer index l; neither denotes a power |
Conventions used in numerical solutions
Vectors are columns unless a question specifies otherwise. A weight matrix has one row per receiving neuron and one column per incoming feature. Write matrix dimensions before multiplying. A bias is added once per receiving neuron or output feature map.
Keep at least six significant digits during intermediate calculations and round the final result as requested. The loss reduction matters: a sum, mean and half-sum produce different gradient factors. The location of ε also matters: a denominator √s + ε differs from √(s + ε).
The sign of a gradient belongs to the derivative, not to the loss value. For example, w = 0.4, η = 0.1 and gradient −2 give the update below.
LaTeX: w_{\mathrm{new}}=0.4-0.1(-2)=0.6In later calculations, state the optimizer convention, initial state and iteration number before substituting values. This prevents mixing momentum velocity with Adam's second-moment accumulator.