# R25 Q2a · Activation, ReLU and numeric outputs

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s4

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s4/

## Question

**Question. Explain why activation functions are needed and sketch ReLU. Is sigmoid appropriate for an unscaled house-price output? Compute softmax([1.2,3.4,0.9]) and sigmoid(2.5).** (10 marks)

## Answer — step by step

An activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary.

![Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation.](https://deep-learning-study-room.vercel.app/assets/figures/v_nonlinear.svg)

Diagram: Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation.. Labels: Two affine layers still make one affine rule | Two affine layers still make one affine rule | Without a nonlinear activation, these two stages can be combined. | Layer 1 and layer 2 | h = 2x + 1 | y = 3h + 4 | Substitute and simplify | y = 3(2x + 1) + 4 | y = 6x + 7 | A nonlinear activation between layers prevents this simple affine collapse.

Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation. Tap the diagram to enlarge.

### Why two linear layers are still linear in the input

For the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second:



$$
y=3(2x+1)+4=6x+3+4=6x+7
$$



This is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent.

| Activation | Output range | Useful property | Main limitation |
| --- | --- | --- | --- |
| Sigmoid | Strictly between 0 and 1 | Useful as a binary-class probability output | Saturates at large positive or negative inputs; gradients become small |
| Tanh | Strictly between −1 and 1 | Can represent signed, zero-centered activations | Also saturates and can contribute to vanishing gradients |
| ReLU | From 0 upward | Simple; derivative is 1 for positive inputs | A unit that stays on the negative side receives zero local gradient |

Sigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score.

**Exam conclusion:** select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction.



$$
\mathrm{ReLU}(z)=\max(0,z)
$$



za

Unscaled house prices are not restricted to(0,1), so a sigmoid cannot directly represent them. Use a linear regression output (or a justified positive-valued transform). A sigmoid could be used only with a deliberately bounded/normalized target and inverse transformation.

### Given

Logits=[1.2,3.4,0.9]; separate sigmoid input=2.5.

### To find

Normalized class probabilities and sigmoid value.

### Formula: which one and why



$$
p_i=\frac{e^{z_i-m}}{\sum_je^{z_j-m}},\quad m=\max_j z_j;\qquad\sigma(z)=\frac1{1+e^{-z}}
$$



### Step 1. Subtract the maximum for stable softmax

[−2.2,0,−2.5]. This common shift leaves softmax unchanged.

### Step 2. Exponentiate then sum

[0.11080315836233387, 1.0, 0.0820849986238988]; sum=1.192888156986.

### Step 3. Divide each by the same sum

| Logit | Exponential / sum | Probability |
| --- | --- | --- |
| 1.2 | 0.110803158362/1.192888156986 | 0.092886460 |
| 3.4 | 1.000000000000/1.192888156986 | 0.838301558 |
| 0.9 | 0.082084998624/1.192888156986 | 0.068811982 |

### Step 4. Compute sigmoid separately

e^(−2.5)=0.082084998624;1+e^(−2.5)=1.082084998624; reciprocal=0.924141820.

### Final answer

Softmax=[0.09288646, 0.838301558, 0.068811982], sum=1.000000000; sigmoid(2.5)=0.924141820.