Deep Learning · TEE · section

Exponentials and natural logarithms

Mathematics revision

Definitions. The exponential function exp(x) = eˣ uses Euler's constant e. The natural logarithm ln is its inverse on positive real values. These functions occur in sigmoid, tanh, softmax and cross-entropy.

LaTeX: e^{-x}=\frac1{e^x},\qquad\ln(e^x)=x

LaTeX: e^{\ln x}=x\ (x>0),\qquad\ln(ab)=\ln a+\ln b\ (a,b>0)

For positive constants and valid domains, exponent and log rules simplify expressions before a calculator is needed. For example, e^(−ln 2) = 1/2. Use natural ln for the loss formulas, not the calculator's base-10 log.

Derivatives used in neural networks

LaTeX: \frac{d}{dz}e^{-z}=-e^{-z},\qquad\frac{d}{dp}[-\ln p]=-\frac1p

The minus sign in the first expression comes from differentiating the inner function −z. Applying the chain rule correctly here gives the sigmoid derivative derived in the activation section.

Brief integration and trigonometry reminder

Integration reverses differentiation up to an additive constant. The basic relevant identities are:

LaTeX: \int x^n\,dx=\frac{x^{n+1}}{n+1}+C\quad(n\ne-1)

LaTeX: \int\frac1x\,dx=\ln|x|+C,\qquad\int e^x\,dx=e^x+C

For angles in radians, d(sin x)/dx = cos x and d(cos x)/dx = −sin x. The identity sin²x + cos²x = 1 concerns ordinary trigonometric functions. Hyperbolic tanh is a different function; using tan instead of tanh gives an incorrect recurrent-network answer.

LaTeX: \tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad\frac{d}{dz}\tanh z=1-\tanh^2z

The supplied MT2 calculation families mainly require derivatives, matrix products, logarithms and elementary statistics. Extended integration or trigonometric exercises are not needed for those families.

Calculator values for activation calculations

At z = 2, sigmoid = 1/(1 + e⁻²) ≈ 0.880797. At z = 1, tanh ≈ 0.761594. Keep unrounded values when a later gradient uses these activations.