# Activation functions and how to calculate them

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s7

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s7/

### What an activation function does, and why it is needed

A neuron first produces the weighted sum z. The activation function f is a rule that transforms that sum into the output a = f(z). Applying an activation does not change the stored weights; it calculates a new output from the current sum.

If every layer only multiplies and adds, several layers still combine into a single affine transformation. A nonlinear activation lets the network represent more complicated relationships. The output activation must also suit the task: sigmoid for a binary-class probability, softmax for mutually exclusive class probabilities, and often a linear output for unrestricted regression.

The **activation value** says what the neuron outputs. The **activation derivative** says how quickly that output changes when z changes, which is needed during backpropagation. For sigmoid at z = 0, the value is 0.5 but the derivative is 0.25. They serve different purposes.

**Yaad rakhne ka cue:** z = activation se pehle ka sum; a = uske baad ka answer; f′ = answer kitni tezi se badlega.

**Definition.** An activation function maps a neuron's preactivation to its output. Nonlinear activations allow a multilayer network to represent nonlinear functions; the output activation also encodes the prediction type. Without a nonlinear activation, a composition of affine layers remains affine.

### Explanation and working



$$
\sigma(z)=\frac{1}{1+e^{-z}},\quad \sigma'(z)=\sigma(z)[1-\sigma(z)]
$$



Sigmoid converts any finite real input into a value strictly between 0 and 1. At zero it is 0.5. It is useful for a binary-class probability or independent multilabel outputs. At very positive or negative inputs it saturates, so its derivative becomes small. Its derivative is at most 0.25.

![Trace the output of each curve as the same raw input moves from negative to positive.](https://deep-learning-study-room.vercel.app/assets/figures/v_activations.svg)

Diagram: Trace the output of each curve as the same raw input moves from negative to positive.. Labels: See what each activation does | See what each activation does | The horizontal axis is the raw score z; the vertical axis is the output. | Sigmoid | 0 | z | 0 < output < 1 | Tanh | 0 | z | −1 < output < 1 | ReLU | 0 | z | Negative → 0; positive → z | The shape of an activation changes both the output and the gradient passed | backward.

Trace the output of each curve as the same raw input moves from negative to positive. Tap the diagram to enlarge.



$$
\tanh(z)=\frac{e^{2z}-1}{e^{2z}+1},\quad \tanh'(z)=1-\tanh^2(z)
$$



Tanh lies between −1 and +1 and is zero-centered. It is common in recurrent hidden states. It also saturates. “Zero-centered” does not mean “no vanishing gradients”.



$$
\operatorname{ReLU}(z)=\max(0,z),\quad f'(z)=\begin{cases}1&z>0\\0&z<0\end{cases}
$$



ReLU returns the input if positive and zero if negative. It is cheap to compute and has derivative 1 on its positive side. Units can become inactive on all relevant inputs, known as dying ReLU. At z = 0 the mathematical derivative is undefined; a convention such as 0 is used in implementations and this course’s exercises when needed.



$$
\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\az&z<0\end{cases},\qquad a=0.01\ \text{unless specified}
$$



The small negative slope a lets some gradient through for negative inputs. At z = −2 and a = 0.01, output = −0.02. Do not take the absolute value. PReLU learns the negative slope; Leaky ReLU normally fixes it.



$$
p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j
$$



Softmax acts on a group of class scores, called logits. All probabilities are positive and sum to 1. Subtracting the same maximum c from all logits leaves the result unchanged and avoids very large exponentials. It is suited to mutually exclusive multiclass output.

**Full softmax example.** For [−1, 0, 1], exponentials are [0.367879, 1, 2.718282]. Sum = 4.086161. Divide each by that same sum: [0.090031, 0.244728, 0.665241]. Check their sum is approximately 1. Three separate sigmoids would give [0.268941, 0.5, 0.731059], which are different and do not sum to 1.

### Derivation of the sigmoid derivative

Write sigmoid as a negative power. The outer derivative contributes −1, and differentiating the inner exponential contributes a second negative sign:



$$
\sigma(z)=(1+e^{-z})^{-1}
$$





$$
\sigma'(z)=-(1+e^{-z})^{-2}(-e^{-z})=\frac{e^{-z}}{(1+e^{-z})^2}
$$



Separate the expression into two factors:



$$
\sigma'(z)=\frac{1}{1+e^{-z}}\frac{e^{-z}}{1+e^{-z}}=\sigma(z)[1-\sigma(z)]
$$



The second factor is 1−σ(z) because subtracting 1/(1+e⁻ᶻ) from 1 leaves e⁻ᶻ/(1+e⁻ᶻ). Once a forward pass has supplied σ(z), its derivative requires no further exponential evaluation.

### Derivation of the tanh derivative

Use the quotient rule on the exponential definition. Here u = e²ᶻ−1 and v = e²ᶻ+1, with u′ = v′ = 2e²ᶻ.



$$
\frac{d}{dz}\tanh z=\frac{2e^{2z}(e^{2z}+1)-2e^{2z}(e^{2z}-1)}{(e^{2z}+1)^2}=\frac{4e^{2z}}{(e^{2z}+1)^2}
$$



The identity 1−tanh²(z) gives the same expression, since the numerator becomes (e²ᶻ+1)²−(e²ᶻ−1)² = 4e²ᶻ. Therefore the backward-pass factor is 1−a² when a = tanh(z).

### Why nonlinear activations are required

For two affine layers without a nonlinear activation:



$$
h=W_1x+b_1,\qquad \hat y=W_2h+b_2=W_2W_1x+(W_2b_1+b_2)
$$



The composition is still one affine transformation. A nonlinear activation between layers changes the family of functions the network can represent. This is the mathematical reason for using nonlinear hidden activations; merely adding affine layers does not provide this expressive advantage.

### sigmoid example 1

**Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.**

### Given

z = -2; activation = sigmoid.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\frac1{1+e^{-z}}
$$





$$
f\prime(z)=a(1-a)
$$



**Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output).

### Solution — step by step

**Step 1: negate the whole input.**



$$
-z=-(-2)=2
$$



**Step 2: calculate the exponential.**



$$
e^{2}\approx7.389056099
$$



**Step 3: add one in the denominator.**



$$
1+7.389056099\approx8.389056099
$$



**Step 4: divide one by the complete denominator.**



$$
a=\frac1{8.389056099}\approx0.119202922
$$



**Step 5: use the activated output in the slope.**



$$
f\prime(-2)=(0.119202922)(1-0.119202922)\approx0.104993585
$$



**Final answer:** a ≈ 0.119202922. Derivative ≈ 0.104993585.

### sigmoid example 2

**Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.**

### Given

z = 0; activation = sigmoid.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\frac1{1+e^{-z}}
$$





$$
f\prime(z)=a(1-a)
$$



**Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output).

### Solution — step by step

**Step 1: negate the whole input.**



$$
-z=-(0)=0
$$



**Step 2: calculate the exponential.**



$$
e^{0}\approx1.000000000
$$



**Step 3: add one in the denominator.**



$$
1+1.000000000\approx2.000000000
$$



**Step 4: divide one by the complete denominator.**



$$
a=\frac1{2.000000000}\approx0.500000000
$$



**Step 5: use the activated output in the slope.**



$$
f\prime(0)=(0.500000000)(1-0.500000000)\approx0.250000000
$$



**Final answer:** a ≈ 0.500000000. Derivative ≈ 0.250000000.

### sigmoid example 3

**Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.**

### Given

z = 2; activation = sigmoid.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\frac1{1+e^{-z}}
$$





$$
f\prime(z)=a(1-a)
$$



**Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output).

### Solution — step by step

**Step 1: negate the whole input.**



$$
-z=-(2)=-2
$$



**Step 2: calculate the exponential.**



$$
e^{-2}\approx0.135335283
$$



**Step 3: add one in the denominator.**



$$
1+0.135335283\approx1.135335283
$$



**Step 4: divide one by the complete denominator.**



$$
a=\frac1{1.135335283}\approx0.880797078
$$



**Step 5: use the activated output in the slope.**



$$
f\prime(2)=(0.880797078)(1-0.880797078)\approx0.104993585
$$



**Final answer:** a ≈ 0.880797078. Derivative ≈ 0.104993585.

### tanh example 1

**Question. For preactivation z = -2, calculate the tanh output and state its derivative.**

### Given

z = -2; activation = tanh.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\frac{e^{2z}-1}{e^{2z}+1}
$$





$$
f\prime(z)=1-a^2
$$



**Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared.

### Solution — step by step

**Step 1: double z before exponentiation.**



$$
2z=2(-2)=-4
$$



**Step 2: calculate the exponential.**



$$
e^{-4}\approx0.018315639
$$



**Step 3: calculate numerator and denominator separately.**



$$
0.018315639-1\approx-0.981684361
$$





$$
0.018315639+1\approx1.018315639
$$



**Step 4: divide.**



$$
a=\frac{-0.981684361}{1.018315639}\approx-0.964027580
$$



**Step 5: square the activated output for its slope.**



$$
f\prime(-2)=1-(-0.964027580)^2\approx0.070650825
$$



**Final answer:** a ≈ -0.964027580. Derivative ≈ 0.070650825.

### tanh example 2

**Question. For preactivation z = 0, calculate the tanh output and state its derivative.**

### Given

z = 0; activation = tanh.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\frac{e^{2z}-1}{e^{2z}+1}
$$





$$
f\prime(z)=1-a^2
$$



**Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared.

### Solution — step by step

**Step 1: double z before exponentiation.**



$$
2z=2(0)=0
$$



**Step 2: calculate the exponential.**



$$
e^{0}\approx1.000000000
$$



**Step 3: calculate numerator and denominator separately.**



$$
1.000000000-1\approx0.000000000
$$





$$
1.000000000+1\approx2.000000000
$$



**Step 4: divide.**



$$
a=\frac{0.000000000}{2.000000000}\approx0.000000000
$$



**Step 5: square the activated output for its slope.**



$$
f\prime(0)=1-(0.000000000)^2\approx1.000000000
$$



**Final answer:** a ≈ 0.000000000. Derivative ≈ 1.000000000.

### tanh example 3

**Question. For preactivation z = 2, calculate the tanh output and state its derivative.**

### Given

z = 2; activation = tanh.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\frac{e^{2z}-1}{e^{2z}+1}
$$





$$
f\prime(z)=1-a^2
$$



**Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared.

### Solution — step by step

**Step 1: double z before exponentiation.**



$$
2z=2(2)=4
$$



**Step 2: calculate the exponential.**



$$
e^{4}\approx54.598150033
$$



**Step 3: calculate numerator and denominator separately.**



$$
54.598150033-1\approx53.598150033
$$





$$
54.598150033+1\approx55.598150033
$$



**Step 4: divide.**



$$
a=\frac{53.598150033}{55.598150033}\approx0.964027580
$$



**Step 5: square the activated output for its slope.**



$$
f\prime(2)=1-(0.964027580)^2\approx0.070650825
$$



**Final answer:** a ≈ 0.964027580. Derivative ≈ 0.070650825.

### ReLU example 1

**Question. For preactivation z = -2, calculate the ReLU output and state its derivative.**

### Given

z = -2; activation = ReLU.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\max(0,z)
$$



**Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

### Solution — step by step

**Step 1: select the branch from the sign of z.**

-2 is negative, so use the negative branch.

**Step 2: calculate the output.**



$$
a=0
$$



**Step 3: state the slope on that branch.**



$$
f\prime(-2)=0
$$



**Final answer:** a ≈ 0.000000000. Derivative ≈ 0.000000000.

### ReLU example 2

**Question. For preactivation z = 0, calculate the ReLU output and state its derivative.**

### Given

z = 0; activation = ReLU.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\max(0,z)
$$



**Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

### Solution — step by step

**Step 1: select the branch from the sign of z.**

Both branches meet at zero. The value is zero, but the left and right slopes differ.

**Step 2: calculate the output.**



$$
a=0
$$



**Step 3: state the slope on that branch.**

The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.

**Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero.

### ReLU example 3

**Question. For preactivation z = 2, calculate the ReLU output and state its derivative.**

### Given

z = 2; activation = ReLU.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\max(0,z)
$$



**Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

### Solution — step by step

**Step 1: select the branch from the sign of z.**

2 is positive, so retain it.

**Step 2: calculate the output.**



$$
a=2
$$



**Step 3: state the slope on that branch.**



$$
f\prime(2)=1
$$



**Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000.

### Leaky ReLU example 1

**Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.**

### Given

z = -2; activation = Leaky ReLU.

α = 0.1.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

### Solution — step by step

**Step 1: select the branch from the sign of z.**

-2 is negative, so use the negative branch.

**Step 2: calculate the output.**



$$
a=-0.2
$$



**Step 3: state the slope on that branch.**



$$
f\prime(-2)=0.1
$$



**Final answer:** a ≈ -0.200000000. Derivative ≈ 0.100000000.

### Leaky ReLU example 2

**Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.**

### Given

z = 0; activation = Leaky ReLU.

α = 0.1.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

### Solution — step by step

**Step 1: select the branch from the sign of z.**

Both branches meet at zero. The value is zero, but the left and right slopes differ.

**Step 2: calculate the output.**



$$
a=0
$$



**Step 3: state the slope on that branch.**

The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.

**Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero.

### Leaky ReLU example 3

**Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.**

### Given

z = 2; activation = Leaky ReLU.

α = 0.1.

### To find

The output a and the derivative with respect to z.

### Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.



$$
a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases}
$$



**Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

### Solution — step by step

**Step 1: select the branch from the sign of z.**

2 is positive, so retain it.

**Step 2: calculate the output.**



$$
a=2
$$



**Step 3: state the slope on that branch.**



$$
f\prime(2)=1
$$



**Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000.

### Softmax for [0, 0, 0]

**Question. Find stable softmax probabilities for logits [0, 0, 0].**

### Given

Logits = [0, 0, 0]; there are three mutually exclusive classes.

### To find

All three class probabilities.

### Formula — which one and why

Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.

A logit is a raw class score. c is the largest score; pᵢ is probability for class i.



$$
c=\max_j z_j
$$





$$
p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}
$$



**Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide.

### Solution — step by step

**Step 1: subtract the largest score from every entry.**



$$
z-c=\begin{bmatrix}0&0&0\end{bmatrix}
$$



**Step 2: exponentiate shifted entry 1.**



$$
e^{0}\approx1.000000000
$$



**Step 3: exponentiate shifted entry 2.**



$$
e^{0}\approx1.000000000
$$



**Step 4: exponentiate shifted entry 3.**



$$
e^{0}\approx1.000000000
$$



**Step 5: add all exponentials into one denominator.**



$$
D=1.000000000+1.000000000+1.000000000\approx3.000000000
$$



**Step 6: divide entry 1 by the same denominator.**



$$
p_1=\frac{1.000000000}{3.000000000}\approx0.333333333
$$



**Step 7: divide entry 2 by the same denominator.**



$$
p_2=\frac{1.000000000}{3.000000000}\approx0.333333333
$$



**Step 8: divide entry 3 by the same denominator.**



$$
p_3=\frac{1.000000000}{3.000000000}\approx0.333333333
$$



**Final answer:** Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding.

### Softmax for [1, 2, 3]

**Question. Find stable softmax probabilities for logits [1, 2, 3].**

### Given

Logits = [1, 2, 3]; there are three mutually exclusive classes.

### To find

All three class probabilities.

### Formula — which one and why

Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.

A logit is a raw class score. c is the largest score; pᵢ is probability for class i.



$$
c=\max_j z_j
$$





$$
p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}
$$



**Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide.

### Solution — step by step

**Step 1: subtract the largest score from every entry.**



$$
z-c=\begin{bmatrix}-2&-1&0\end{bmatrix}
$$



**Step 2: exponentiate shifted entry 1.**



$$
e^{-2}\approx0.135335283
$$



**Step 3: exponentiate shifted entry 2.**



$$
e^{-1}\approx0.367879441
$$



**Step 4: exponentiate shifted entry 3.**



$$
e^{0}\approx1.000000000
$$



**Step 5: add all exponentials into one denominator.**



$$
D=0.135335283+0.367879441+1.000000000\approx1.503214724
$$



**Step 6: divide entry 1 by the same denominator.**



$$
p_1=\frac{0.135335283}{1.503214724}\approx0.090030573
$$



**Step 7: divide entry 2 by the same denominator.**



$$
p_2=\frac{0.367879441}{1.503214724}\approx0.244728471
$$



**Step 8: divide entry 3 by the same denominator.**



$$
p_3=\frac{1.000000000}{1.503214724}\approx0.665240956
$$



**Final answer:** Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding.