# Model answer: why a network needs nonlinear activation

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s8

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s8/

## Question

**Question.** Why are activation functions required in a multilayer neural network? Compare sigmoid, tanh and ReLU, including their limitations.

## Answer — step by step

An activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary.

![Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation.](https://deep-learning-study-room.vercel.app/assets/figures/v_nonlinear.svg)

Diagram: Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation.. Labels: Two affine layers still make one affine rule | Two affine layers still make one affine rule | Without a nonlinear activation, these two stages can be combined. | Layer 1 and layer 2 | h = 2x + 1 | y = 3h + 4 | Substitute and simplify | y = 3(2x + 1) + 4 | y = 6x + 7 | A nonlinear activation between layers prevents this simple affine collapse.

Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation. Tap the diagram to enlarge.

### Why two linear layers are still linear in the input

For the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second:



$$
y=3(2x+1)+4=6x+3+4=6x+7
$$



This is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent.

| Activation | Output range | Useful property | Main limitation |
| --- | --- | --- | --- |
| Sigmoid | Strictly between 0 and 1 | Useful as a binary-class probability output | Saturates at large positive or negative inputs; gradients become small |
| Tanh | Strictly between −1 and 1 | Can represent signed, zero-centered activations | Also saturates and can contribute to vanishing gradients |
| ReLU | From 0 upward | Simple; derivative is 1 for positive inputs | A unit that stays on the negative side receives zero local gradient |

Sigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score.

**Exam conclusion:** select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction.