Deep Learning · TEE · formulas

Tanh and its derivative

Tanh can produce a negative or positive output. In the formula, calculate the exponential first, then form the top and bottom of the fraction separately. The derivative uses the activated answer, not the raw input.

\tanh(z)=\frac{e^{2z}-1}{e^{2z}+1}LaTeX: \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1}
\tanh\prime(z)=1-\tanh^2(z)LaTeX: \tanh\prime(z)=1-\tanh^2(z)

Question. Calculate tanh(0.5) and its derivative with respect to the input.

Given

z = 0.5.

To find

Find tanh(z) and its derivative.

Formula — which one and why

Tanh and its slope. Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a².

z = value entering tanh; a = its output. A prime means derivative, or local rate of change.

a=\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}}LaTeX: a=\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}}
\frac{da}{dz}=1-a^2LaTeX: \frac{da}{dz}=1-a^2

Yaad rakhne ka cue: Tanh signed output deta hai; slope = 1 minus output ka square.

Solution — step by step

Step 1: identify the input. The input z is 0.5. In this version of the tanh formula, the exponential uses 2z.

2z=2(0.5)=1,\qquad e^{2z}=e^1\approx2.718281828LaTeX: 2z=2(0.5)=1,\qquad e^{2z}=e^1\approx2.718281828

Step 2: calculate the numerator. Subtract 1 from that exponential.

e^{2z}-1=2.718281828-1=1.718281828LaTeX: e^{2z}-1=2.718281828-1=1.718281828

Step 3: calculate the denominator. Add 1 to the same exponential.

e^{2z}+1=2.718281828+1=3.718281828LaTeX: e^{2z}+1=2.718281828+1=3.718281828

Step 4: divide the top by the bottom.

a=\tanh(0.5)=\frac{1.718281828}{3.718281828}\approx0.462117157LaTeX: a=\tanh(0.5)=\frac{1.718281828}{3.718281828}\approx0.462117157

Step 5: calculate the local derivative. Square the output a and subtract it from 1. Keep the calculator's unrounded result.

a^2\approx0.213552267,\qquad1-a^2\approx0.786447733LaTeX: a^2\approx0.213552267,\qquad1-a^2\approx0.786447733

Final answer: tanh output ≈ 0.462117157; local derivative ≈ 0.786447733. The output becomes the next layer's input; the derivative is used when carrying a gradient backward.

tanh example 1

Question. For preactivation z = -2, calculate the tanh output and state its derivative.

Given

z = -2; activation = tanh.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

a=\frac{e^{2z}-1}{e^{2z}+1}LaTeX: a=\frac{e^{2z}-1}{e^{2z}+1}
f\prime(z)=1-a^2LaTeX: f\prime(z)=1-a^2

Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.

Solution — step by step

Step 1: double z before exponentiation.

2z=2(-2)=-4LaTeX: 2z=2(-2)=-4

Step 2: calculate the exponential.

e^{-4}\approx0.018315639LaTeX: e^{-4}\approx0.018315639

Step 3: calculate numerator and denominator separately.

0.018315639-1\approx-0.981684361LaTeX: 0.018315639-1\approx-0.981684361
0.018315639+1\approx1.018315639LaTeX: 0.018315639+1\approx1.018315639

Step 4: divide.

a=\frac{-0.981684361}{1.018315639}\approx-0.964027580LaTeX: a=\frac{-0.981684361}{1.018315639}\approx-0.964027580

Step 5: square the activated output for its slope.

f\prime(-2)=1-(-0.964027580)^2\approx0.070650825LaTeX: f\prime(-2)=1-(-0.964027580)^2\approx0.070650825

Final answer: a ≈ -0.964027580. Derivative ≈ 0.070650825.

tanh example 2

Question. For preactivation z = 0, calculate the tanh output and state its derivative.

Given

z = 0; activation = tanh.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

a=\frac{e^{2z}-1}{e^{2z}+1}LaTeX: a=\frac{e^{2z}-1}{e^{2z}+1}
f\prime(z)=1-a^2LaTeX: f\prime(z)=1-a^2

Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.

Solution — step by step

Step 1: double z before exponentiation.

2z=2(0)=0LaTeX: 2z=2(0)=0

Step 2: calculate the exponential.

e^{0}\approx1.000000000LaTeX: e^{0}\approx1.000000000

Step 3: calculate numerator and denominator separately.

1.000000000-1\approx0.000000000LaTeX: 1.000000000-1\approx0.000000000
1.000000000+1\approx2.000000000LaTeX: 1.000000000+1\approx2.000000000

Step 4: divide.

a=\frac{0.000000000}{2.000000000}\approx0.000000000LaTeX: a=\frac{0.000000000}{2.000000000}\approx0.000000000

Step 5: square the activated output for its slope.

f\prime(0)=1-(0.000000000)^2\approx1.000000000LaTeX: f\prime(0)=1-(0.000000000)^2\approx1.000000000

Final answer: a ≈ 0.000000000. Derivative ≈ 1.000000000.

tanh example 3

Question. For preactivation z = 2, calculate the tanh output and state its derivative.

Given

z = 2; activation = tanh.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

a=\frac{e^{2z}-1}{e^{2z}+1}LaTeX: a=\frac{e^{2z}-1}{e^{2z}+1}
f\prime(z)=1-a^2LaTeX: f\prime(z)=1-a^2

Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.

Solution — step by step

Step 1: double z before exponentiation.

2z=2(2)=4LaTeX: 2z=2(2)=4

Step 2: calculate the exponential.

e^{4}\approx54.598150033LaTeX: e^{4}\approx54.598150033

Step 3: calculate numerator and denominator separately.

54.598150033-1\approx53.598150033LaTeX: 54.598150033-1\approx53.598150033
54.598150033+1\approx55.598150033LaTeX: 54.598150033+1\approx55.598150033

Step 4: divide.

a=\frac{53.598150033}{55.598150033}\approx0.964027580LaTeX: a=\frac{53.598150033}{55.598150033}\approx0.964027580

Step 5: square the activated output for its slope.

f\prime(2)=1-(0.964027580)^2\approx0.070650825LaTeX: f\prime(2)=1-(0.964027580)^2\approx0.070650825

Final answer: a ≈ 0.964027580. Derivative ≈ 0.070650825.