# Drill 10 Sigmoid neuron gradient

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s10

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s10/

## Question

**Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update.

## Answer — step by step

### Given

x = 2; w = 0.5; b = 0; target y = 1.

Activation: sigmoid; loss: half squared error.

Learning rate η = 0.1.

### To find

Find the output, delta, weight/bias gradients and one update.

### Formula — which one and why

**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.



$$
z=\sum_iw_ix_i+b
$$





$$
a=f(z)
$$



**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.

**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.



$$
p=\frac{1}{1+e^{-z}}
$$





$$
\frac{dp}{dz}=p(1-p)
$$



**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).

**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.



$$
L=\frac12\sum_j(o_j-y_j)^2
$$





$$
\frac{\partial L}{\partial o_j}=o_j-y_j
$$



**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.

**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.



$$
\delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})
$$





$$
\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}
$$





$$
\frac{\partial L}{\partial W_{ji}}=\delta_ja_i
$$





$$
\frac{\partial L}{\partial b_j}=\delta_j
$$





$$
W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}
$$



**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.

### Solution — step by step

### Meaning and formulas

Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly.



$$
z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2
$$



The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z.



$$
\delta=(p-y)p(1-p)
$$





$$
\frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta
$$





$$
w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta
$$



### Step 1: weighted sum and sigmoid



$$
z=0.5\times2+0=1
$$





$$
p=\frac1{1+e^{-1}}=0.731058579
$$



### Step 2: error and activation slope



$$
p-y=0.731058579-1=-0.268941421
$$





$$
p(1-p)=0.731058579\times0.268941421=0.196611933
$$



### Step 3: multiply the linked effects to obtain delta



$$
\delta=(-0.268941421)\times0.196611933=-0.052877093
$$



### Step 4: multiply delta by the input to each parameter

The weight receives x = 2. The bias effectively receives the constant input 1.



$$
\frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186
$$





$$
\frac{\partial L}{\partial b}=-0.052877093
$$



### Step 5: subtract learning rate times each gradient



$$
w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419
$$





$$
b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709
$$



**Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here.