# Power, constant and sum derivatives

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/3

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/3/

Derivatives describe local rates of change. Partial derivatives hold other independent variables fixed; their collection forms the gradient used for optimization.



$$
\frac{d}{dx}(cx^n)=cnx^{n-1}
$$





$$
\frac{d}{dx}c=0
$$





$$
(f+g)\prime=f\prime+g\prime
$$



## Derivatives without a calculus course

**Definition.** A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector.



$$
\nabla_\theta J=\begin{bmatrix}\partial J/\partial\theta_1\\\vdots\\\partial J/\partial\theta_d\end{bmatrix}
$$



The rules needed for the supplied neural-network numericals are:

| Function or operation | Derivative |
| --- | --- |
| Constant c | 0 |
| xⁿ | nxⁿ⁻¹ |
| c f(x) | c f′(x) |
| f(x) + g(x) | f′(x) + g′(x) |
| f(x)g(x) | f′(x)g(x) + f(x)g′(x) |
| eˣ | eˣ |
| ln x, x > 0 | 1/x |
| f(g(x)) | f′(g(x))g′(x) |

### A parameter derivative is not an input derivative

For z = wx + b, the variable named in the denominator determines which derivative is required.



$$
\frac{\partial z}{\partial w}=x,\qquad\frac{\partial z}{\partial x}=w,\qquad\frac{\partial z}{\partial b}=1
$$



At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities.

### Gradient direction

For a small parameter displacement Δθ, first-order Taylor expansion gives:



$$
J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta
$$



The gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large.

### Differentiate before substituting: 3w^2+5

**Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.**

### Given

J(w) = 3w^2+5; w = 2.

### To find

The function value and its derivative value.

### Formula — which one and why

The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.

c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.



$$
\frac{d(cw^n+k)}{dw}=cnw^{n-1}
$$



**Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero.

### Solution — step by step

**Step 1: evaluate the function itself.**



$$
J(2)=3(2)^2+5=17
$$



**Step 2: differentiate symbolically.**



$$
J\prime(w)=3(2)w^{2-1}+0
$$



**Step 3: put the given w into the derivative.**



$$
J\prime(2)=3(2)(2)^{1}=12
$$



**Final answer:** J = 17; derivative = 12.

### Differentiate before substituting: 0.7w^3+0

**Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.**

### Given

J(w) = 0.7w^3+0; w = 1.

### To find

The function value and its derivative value.

### Formula — which one and why

The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.

c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.



$$
\frac{d(cw^n+k)}{dw}=cnw^{n-1}
$$



**Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero.

### Solution — step by step

**Step 1: evaluate the function itself.**



$$
J(1)=0.7(1)^3+0=0.7
$$



**Step 2: differentiate symbolically.**



$$
J\prime(w)=0.7(3)w^{3-1}+0
$$



**Step 3: put the given w into the derivative.**



$$
J\prime(1)=0.7(3)(1)^{2}=2.1
$$



**Final answer:** J = 0.7; derivative = 2.1.

### Differentiate before substituting: 2w^1+4

**Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.**

### Given

J(w) = 2w^1+4; w = -1.

### To find

The function value and its derivative value.

### Formula — which one and why

The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.

c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.



$$
\frac{d(cw^n+k)}{dw}=cnw^{n-1}
$$



**Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero.

### Solution — step by step

**Step 1: evaluate the function itself.**



$$
J(-1)=2(-1)^1+4=2
$$



**Step 2: differentiate symbolically.**



$$
J\prime(w)=2(1)w^{1-1}+0
$$



**Step 3: put the given w into the derivative.**



$$
J\prime(-1)=2(1)(-1)^{0}=2
$$



**Final answer:** J = 2; derivative = 2.