Deep Learning · TEE · section

R25 Q2a · Activation, ReLU and numeric outputs

February 2025 re-exam · complete solved paper

Question

Question. Explain why activation functions are needed and sketch ReLU. Is sigmoid appropriate for an unscaled house-price output? Compute softmax([1.2,3.4,0.9]) and sigmoid(2.5). (10 marks)

Answer — step by step

An activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary.

Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation. Tap the diagram to enlarge.

Why two linear layers are still linear in the input

For the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second:

This is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent.

Activation	Output range	Useful property	Main limitation
Sigmoid	Strictly between 0 and 1	Useful as a binary-class probability output	Saturates at large positive or negative inputs; gradients become small
Tanh	Strictly between −1 and 1	Can represent signed, zero-centered activations	Also saturates and can contribute to vanishing gradients
ReLU	From 0 upward	Simple; derivative is 1 for positive inputs	A unit that stays on the negative side receives zero local gradient

Sigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score.

Exam conclusion: select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction.

$\mathrm{ReLU}(z)=\max(0,z)$ LaTeX: \mathrm{ReLU}(z)=\max(0,z)

Unscaled house prices are not restricted to(0,1), so a sigmoid cannot directly represent them. Use a linear regression output (or a justified positive-valued transform). A sigmoid could be used only with a deliberately bounded/normalized target and inverse transformation.

Given

Logits=[1.2,3.4,0.9]; separate sigmoid input=2.5.

To find

Normalized class probabilities and sigmoid value.

Formula: which one and why

$p_i=\frac{e^{z_i-m}}{\sum_je^{z_j-m}},\quad m=\max_j z_j;\qquad\sigma(z)=\frac1{1+e^{-z}}$ LaTeX: p_i=\frac{e^{z_i-m}}{\sum_je^{z_j-m}},\quad m=\max_j z_j;\qquad\sigma(z)=\frac1{1+e^{-z}}

Step 1. Subtract the maximum for stable softmax

[−2.2,0,−2.5]. This common shift leaves softmax unchanged.

Step 2. Exponentiate then sum

[0.11080315836233387, 1.0, 0.0820849986238988]; sum=1.192888156986.

Step 3. Divide each by the same sum

Logit	Exponential / sum	Probability
1.2	0.110803158362/1.192888156986	0.092886460
3.4	1.000000000000/1.192888156986	0.838301558
0.9	0.082084998624/1.192888156986	0.068811982

Step 4. Compute sigmoid separately

e^(−2.5)=0.082084998624;1+e^(−2.5)=1.082084998624; reciprocal=0.924141820.

Final answer

Softmax=[0.09288646, 0.838301558, 0.068811982], sum=1.000000000; sigmoid(2.5)=0.924141820.