Deep Learning · MT2 · section

Model answer: why a network needs nonlinear activation

Neural-network foundations

Question

Question. Why are activation functions required in a multilayer neural network? Compare sigmoid, tanh and ReLU, including their limitations.

Answer — step by step

An activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary.

Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation. Tap the diagram to enlarge.

Why two linear layers are still linear in the input

For the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second:

This is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent.

Activation	Output range	Useful property	Main limitation
Sigmoid	Strictly between 0 and 1	Useful as a binary-class probability output	Saturates at large positive or negative inputs; gradients become small
Tanh	Strictly between −1 and 1	Can represent signed, zero-centered activations	Also saturates and can contribute to vanishing gradients
ReLU	From 0 upward	Simple; derivative is 1 for positive inputs	A unit that stays on the negative side receives zero local gradient

Sigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score.

Exam conclusion: select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction.