Deep Learning · TEE · section
Derivatives without a calculus course
The maths behind a network
Definition. A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector.
LaTeX: \nabla_\theta J=\begin{bmatrix}\partial J/\partial\theta_1\\\vdots\\\partial J/\partial\theta_d\end{bmatrix}The rules needed for the supplied neural-network numericals are:
| Function or operation | Derivative |
|---|---|
| Constant c | 0 |
| xⁿ | nxⁿ⁻¹ |
| c f(x) | c f′(x) |
| f(x) + g(x) | f′(x) + g′(x) |
| f(x)g(x) | f′(x)g(x) + f(x)g′(x) |
| eˣ | eˣ |
| ln x, x > 0 | 1/x |
| f(g(x)) | f′(g(x))g′(x) |
A parameter derivative is not an input derivative
For z = wx + b, the variable named in the denominator determines which derivative is required.
LaTeX: \frac{\partial z}{\partial w}=x,\qquad\frac{\partial z}{\partial x}=w,\qquad\frac{\partial z}{\partial b}=1At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities.
Gradient direction
For a small parameter displacement Δθ, first-order Taylor expansion gives:
LaTeX: J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\thetaThe gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large.
Differentiate before substituting: 3w^2+5
Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.
Given
J(w) = 3w^2+5; w = 2.
To find
The function value and its derivative value.
Formula — which one and why
The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.
c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.
LaTeX: \frac{d(cw^n+k)}{dw}=cnw^{n-1}Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.
Solution — step by step
Step 1: evaluate the function itself.
LaTeX: J(2)=3(2)^2+5=17Step 2: differentiate symbolically.
LaTeX: J\prime(w)=3(2)w^{2-1}+0Step 3: put the given w into the derivative.
LaTeX: J\prime(2)=3(2)(2)^{1}=12Final answer: J = 17; derivative = 12.
Differentiate before substituting: 0.7w^3+0
Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.
Given
J(w) = 0.7w^3+0; w = 1.
To find
The function value and its derivative value.
Formula — which one and why
The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.
c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.
LaTeX: \frac{d(cw^n+k)}{dw}=cnw^{n-1}Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.
Solution — step by step
Step 1: evaluate the function itself.
LaTeX: J(1)=0.7(1)^3+0=0.7Step 2: differentiate symbolically.
LaTeX: J\prime(w)=0.7(3)w^{3-1}+0Step 3: put the given w into the derivative.
LaTeX: J\prime(1)=0.7(3)(1)^{2}=2.1Final answer: J = 0.7; derivative = 2.1.
Differentiate before substituting: 2w^1+4
Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.
Given
J(w) = 2w^1+4; w = -1.
To find
The function value and its derivative value.
Formula — which one and why
The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.
c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.
LaTeX: \frac{d(cw^n+k)}{dw}=cnw^{n-1}Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.
Solution — step by step
Step 1: evaluate the function itself.
LaTeX: J(-1)=2(-1)^1+4=2Step 2: differentiate symbolically.
LaTeX: J\prime(w)=2(1)w^{1-1}+0Step 3: put the given w into the derivative.
LaTeX: J\prime(-1)=2(1)(-1)^{0}=2Final answer: J = 2; derivative = 2.