Deep Learning · MT2 · section

All formulas: meanings, use and substitution

Formula reference and exam revision

These are the formulas used by this course. Each entry names its symbols and the type of calculation it serves. Numerical practice in the teaching chapters shows the longer working. Follow any different convention explicitly stated in the examination.

Formula 1: Mean, variance and standard deviation

LaTeX: \mu=\frac{1}{n}\sum_i x_i,\quad v=\frac{1}{n}\sum_i(x_i-\mu)^2,\quad s=\sqrt{v}

Symbols: n is the count; mu is the mean; v is the population variance; s is the standard deviation.

Use: Use when calculating a population mean/variance or standardization. Sample-variance estimates may use n−1 when explicitly requested.

Substitution example: For [2,4]: mean = 3; deviations = −1,1; squares = 1,1; variance = 1; standard deviation = 1.

Formula 2: Standardization

LaTeX: x_{scaled}=\frac{x-\mu_{train}}{s_{train}}

Symbols: Use the mean and standard deviation estimated from training data.

Use: Use to express a value relative to the training distribution. The standard deviation must be nonzero.

Substitution example: For x = 7, mean = 5, standard deviation = 2: (7−5)/2 = 1.

Formula 3: Power, constant and sum derivatives

LaTeX: \frac{d}{dx}(cx^n)=cnx^{n-1},\quad\frac{d}{dx}c=0,\quad (f+g)\prime=f\prime+g\prime

Symbols: c is a constant, x is the changing variable and n is the power.

Use: Use to differentiate the loss supplied in a gradient-descent question.

Substitution example: For 0.7x³: derivative = 0.7 × 3 × x² = 2.1x².

Formula 4: Chain rule

LaTeX: \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw}

Symbols: z is the intermediate value linking w to L.

Use: Multiply derivatives along a path; add contributions from distinct paths that meet.

Substitution example: For z = 3w+1 and L = z² at w = 1: z = 4; derivative = (2×4)×3 = 24.

Formula 5: Dot product and a neuron

LaTeX: z=\sum_i w_i x_i+b,\qquad a=f(z)

Symbols: x is the input; w is its matching weight; b is one bias; z is the weighted sum; a is the activated output.

Use: Multiply each matching pair, add the products, add bias once, then activate.

Substitution example: x = [2,1], w = [0.5,−0.2], b = 0.1: z = 1−0.2+0.1 = 0.9.

Formula 6: Matrix multiplication

LaTeX: (AB)_{ij}=\sum_k A_{ik}B_{kj}

Symbols: i selects a row of A; j selects a column of B; k runs along the matching entries.

Use: Use row-times-column sums. A with shape m×n and B with n×p produces m×p.

Substitution example: Row [1,2] times column [3,4] gives 1×3+2×4 = 11.

Formula 7: Layer forward pass

LaTeX: z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)})

Symbols: l labels the layer; the previous layer activations become inputs to the next layer.

Use: For receiving-neuron rows, compute each row dot product and add that receiving neuron’s bias. Finish the whole layer before moving on.

Formula 8: Sigmoid and its derivative

LaTeX: \sigma(z)=\frac{1}{1+e^{-z}},\quad \sigma\prime(z)=\sigma(z)[1-\sigma(z)]

Symbols: e is approximately 2.71828; sigma is the output.

Use: Use for a sigmoid forward pass and for its local backpropagation derivative.

Substitution example: At z = 0: output = 1/(1+1) = 0.5; derivative = 0.5×0.5 = 0.25.

Formula 9: Tanh and its derivative

LaTeX: \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1},\quad \tanh\prime(z)=1-\tanh^2(z)

Symbols: Tanh gives an output between −1 and 1.

Use: Use for a stated tanh activation, commonly an RNN hidden state or recurrent candidate.

Substitution example: At z = 0: numerator = 0, denominator = 2; output = 0; derivative = 1.

Formula 10: ReLU and its derivative

LaTeX: f(z)=\max(0,z),\quad f\prime(z)=\begin{cases}1&z>0\\0&z<0\end{cases}

Symbols: max means choose the larger value.

Use: Use for ReLU. At zero the derivative is mathematically undefined; use the convention stated by the question or implementation.

Substitution example: For −2, output = 0 and derivative = 0. For 3, output = 3 and derivative = 1.

Formula 11: Leaky ReLU

LaTeX: f(z)=\begin{cases}z&z\geq0\\az&z<0\end{cases}

Symbols: a is the small negative-side slope, commonly 0.01 unless specified.

Use: For negative z multiply by a; the output remains negative.

Substitution example: For z = −2 and a = 0.01: output = −0.02; derivative on this side is 0.01.

Formula 12: Stable softmax

LaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_j z_j

Symbols: z values are class scores; p values are probabilities.

Use: Subtract the same maximum from every score, exponentiate, add, divide each by that total.

Substitution example: For [0,0]: exponentials = [1,1]; total = 2; probabilities = [0.5,0.5].

Formula 13: Squared and absolute losses

LaTeX: \mathrm{SSE}=\sum_i(o_i-y_i)^2

LaTeX: L_{\mathrm{half}}=\frac{\mathrm{SSE}}2

LaTeX: L_{\mathrm{MSE}}=\frac{\mathrm{SSE}}n

LaTeX: L_{\mathrm{MAE}}=\frac1n\sum_i|o_i-y_i|

Symbols: o is prediction; y is target; n is the number of averaged terms.

Use: Read whether the question uses a half sum, a mean, or an unhalved sum. The scaling changes gradients.

Substitution example: For errors [−1,2]: half sum = (1+4)/2 = 2.5; MSE = 2.5; MAE = (1+2)/2 = 1.5. The first two coincide only because n = 2.

Formula 14: Binary cross entropy

LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

Symbols: y is 0 or 1; p is the predicted probability for class 1; ln is the natural logarithm.

Use: Use for a sigmoid binary prediction. If y = 1 use −ln(p); if y = 0 use −ln(1−p).

Substitution example: For y = 1 and p = 0.8: loss = 0.223144.

Formula 15: Categorical cross entropy

LaTeX: H(p,q)=-\sum_i p_i\ln q_i,\qquad\log_2q=\frac{\ln q}{\ln2}

For a soft target distribution, retain every nonzero target-weighted term. Natural logs give nats; base-2 logs give bits.

LaTeX: L=-\sum_i y_i\ln p_i=-\ln p_{true}

Symbols: The final equality assumes one-hot labels, with exactly one y equal to 1.

Use: Use for mutually exclusive multiclass labels. Select the probability of the true class.

Substitution example: True class 2 and probabilities [0.1,0.8,0.1]: loss = −ln(0.8) = 0.223144.

Formula 16: Output delta for half squared loss

LaTeX: \delta^{out}=(o-y)f\prime(z)

Symbols: Delta is the loss derivative with respect to the preactivation z.

Use: Use with half squared loss. For unhalved one-output MSE multiply by 2; for an MSE averaged over m outputs multiply the half-sum delta by 2/m.

Substitution example: Sigmoid output 0.5 and target 1: delta = (−0.5)×0.25 = −0.125.

Formula 17: Sigmoid plus BCE; softmax plus CCE

LaTeX: \delta^{out}=p-y

Symbols: p and y can be vectors for softmax.

Use: This is the combined derivative with respect to the logits. Do not multiply by the activation derivative again.

Substitution example: Binary p = 0.8, y = 1: delta = −0.2.

Formula 18: Hidden delta, weight gradient and bias gradient

LaTeX: \delta_j^h=f\prime(z_j^h)\sum_k w_{kj}^{out}\delta_k^{out},\quad\frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\quad\frac{\partial L}{\partial b_j}=\delta_j

Symbols: a_i is the activation sending a signal along the weight; delta_j belongs to the receiving neuron.

Use: Use OLD downstream weights throughout one backward pass. Add all outgoing-path contributions.

Substitution example: If receiving delta = −0.2 and sending activation = 3: weight gradient = −0.6; bias gradient = −0.2.

Formula 19: Gradient descent

LaTeX: \theta_{new}=\theta-\eta g,\qquad g=\frac{\partial J}{\partial\theta}

Symbols: theta is the parameter; eta is learning rate; g is the current gradient.

Use: Recalculate g at the current parameter unless the question supplies gradients directly.

Substitution example: theta = 1, eta = 0.1, g = −2 gives 1−0.1(−2) = 1.2.

Formula 20: Batch and mini-batch gradient

LaTeX: g_{batch}=\frac1B\sum_{i=1}^{B}g_i,\quad U_{epoch}=\left\lceil\frac NB\right\rceil

Symbols: B is batch size; N is sample count; U is updates per epoch when the final incomplete batch is retained.

Use: SGD uses B = 1. Full batch uses B = N. Average only when the stated objective is a mean.

Substitution example: N = 105, B = 20: six updates, with five samples in the last batch.

Formula 21: Momentum: signed velocity

LaTeX: v_t=\beta v_{t-1}-\eta g_t,\qquad\theta_t=\theta_{t-1}+v_t

Symbols: v stores signed movement; beta retains part of old movement.

Use: Use this convention consistently. Other textbooks may define an unsigned gradient accumulator and subtract it instead.

Substitution example: beta = 0.8, old v = −0.2, eta = 0.1, g = 1.8 gives v = −0.34.

Formula 22: Nesterov: look-ahead gradient

LaTeX: \theta_{look}=\theta+\beta v,\quad v_{new}=\beta v-\eta g(\theta_{look}),\quad\theta_{new}=\theta+v_{new}

Symbols: The look-ahead point is temporary; the update is added to the original theta.

Use: Evaluate the gradient at the look-ahead point, not at the old theta.

Substitution example: theta = 1.8, beta = 0.8 and v = −0.2 gives look-ahead = 1.64.

Formula 23: AdaGrad

LaTeX: G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon}

Symbols: G is accumulated squared gradient; epsilon prevents division by zero.

Use: Accumulate every squared gradient, separately for each parameter. Use the epsilon placement stated by the problem.

Substitution example: Old G = 4 and g = −3 gives G = 13.

Formula 24: RMSProp

LaTeX: s_t=\rho s_{t-1}+(1-\rho)g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon}

Symbols: rho is the decay; s is a decaying mean of squared gradients.

Use: Calculate the new square memory first, then update the parameter.

Substitution example: rho = 0.9, old s = 4, g = −3 gives s = 3.6+0.9 = 4.5.

Formula 25: Adam: both memories

LaTeX: m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2

Symbols: m remembers signed gradients; v remembers squared gradients.

Use: Start with the initial states specified. Common defaults are beta1 = 0.9 and beta2 = 0.999.

Formula 26: Adam: correction and update

LaTeX: \widehat m_t=\frac{m_t}{1-\beta_1^t},\quad\widehat v_t=\frac{v_t}{1-\beta_2^t},\quad\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}

Symbols: t counts updates starting at 1, not 0.

Use: Correct both zero-initialized memories before the update; do not multiply by the gradient a second time.

Substitution example: At t = 1, m = 0.2 and beta1 = 0.9: corrected m = 0.2/0.1 = 2.

Formula 27: L1 penalty and update

LaTeX: J=L+\lambda\sum_j|w_j|,\quad w_{new}=w-\eta\left[g_{data}+\lambda\operatorname{sign}(w)\right]

Symbols: lambda is regularization strength; sign(w) is +1 for positive w and −1 for negative w.

Use: At zero, L1 has a subgradient rather than a unique derivative; use the specified convention. Include only parameters named in the penalty.

Substitution example: w = −2, lambda = 0.1, data gradient = 0 gives penalty gradient = −0.1.

Formula 28: L2 penalty and update with a half

LaTeX: J=L+\frac\lambda2\sum_jw_j^2,\quad w_{new}=w-\eta(g_{data}+\lambda w)

Symbols: The half cancels the 2 produced by differentiating a square.

Use: If the objective uses lambda times the sum of squares WITHOUT a half, use 2 lambda w instead.

Substitution example: w = 2, lambda = 0.1, data gradient = 0.3 gives total gradient = 0.5.

Formula 29: Inverted dropout

LaTeX: q=1-p,\quad a_{train}=\frac{m\odot a}{q},\quad a_{test}=a

Symbols: p is drop probability; q is keep probability; mask m contains zeros and ones; the circled dot means elementwise multiplication.

Use: During training, zero dropped entries and divide kept entries by q. Disable dropout at inference.

Substitution example: a = 3, q = 0.5, mask = 1 gives 3/0.5 = 6.

Formula 30: Convolution output dimensions

LaTeX: N_{out}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1

Symbols: N is input height or width; P is padding on each side; K is kernel size; S is stride; floor rounds down.

Use: Apply separately to height and width. Output channels equal the number of filters. This formula assumes dilation 1.

Substitution example: N = 7, P = 1, K = 3, S = 2 gives floor(6/2)+1 = 4.

Formula 31: Convolution value

LaTeX: Y_{ij,o}=\sum_c\sum_u\sum_v K_{uvco}X_{iS+u-P,jS+v-P,c}+b_o

Symbols: c runs over input channels; u,v run over kernel positions; o selects one filter.

Use: This is the usual deep-learning cross-correlation operation. Use zeros outside the padded image. Multiply matching cells, add all channels, then add the filter bias once.

Formula 32: Max, min and average pooling

Min pooling keeps the smallest entry in each window. For a four-entry window:

LaTeX: y_{\min}=\min(x_1,x_2,x_3,x_4)

LaTeX: y_{max}=\max_{(i,j)\in W}x_{ij},\quad y_{avg}=\frac1{|W|}\sum_{(i,j)\in W}x_{ij}

Symbols: W is one pooling window; its size is the number of included cells.

Use: Move the window by the stated stride; process channels separately. For padding, follow the stated average-pooling denominator convention.

Substitution example: For [1,4,2,3]: maximum = 4; average = 10/4 = 2.5.

Formula 33: Convolution parameter count

LaTeX: P_{conv}=(K_hK_wC_{in}+1)C_{out}

Symbols: Kernel height and width times input channels gives weights per filter; +1 is one bias; Cout is filter count.

Use: Use for a fully connected, ungrouped convolution with bias. Output height and width do not multiply the parameter count.

Substitution example: 3×3 kernel, 3 input channels, 2 filters: (27+1)×2 = 56.

Formula 34: Flatten and dense parameter count

LaTeX: N_{flat}=HWC,\qquad P_{dense}=(N_{in}+1)N_{out}

Symbols: H,W,C are activation dimensions; Nin and Nout are dense-layer input and output counts.

Use: Flatten has no trainable parameters. Dense biases belong to receiving neurons.

Substitution example: 2×2×2 flattens to 8; dense 3 then has (8+1)×3 = 27 parameters.

Formula 35: Convolution multiply-accumulate count

LaTeX: MAC=H_{out}W_{out}K_hK_wC_{in}C_{out}

Symbols: Each output value uses kernel area times input-channel count multiplications.

Use: Used for Inception bottleneck comparisons. This counts one multiply-accumulate as one MAC and ignores bias additions. Some FLOP conventions count multiply and add separately.

Formula 36: Concatenation and residual addition

LaTeX: C_{concat}=\sum_b C_b,\qquad y_{res}=F(x)+x

Symbols: b labels branches with matching spatial dimensions.

Use: Inception concatenates channels; residual addition requires matching full shapes. Use a projection when needed.

Substitution example: Channels 16+32+8+8 give 64. Adding two 14×14×64 tensors keeps 14×14×64.

Formula 37: Vanilla RNN

LaTeX: h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad o_t=W_{hy}h_t+b_y

Symbols: x_t is current input; h_previous is previous state; the same matrices are reused at every time step.

Use: Calculate the two matrix-vector products, add bias, then tanh. Apply the stated output activation to o_t when required.

Formula 46: Batch normalization and learned scale/shift

Use during training to normalize intermediate activations before a learned scale and shift. m is the number of values; μ_B and σ²_B are the batch mean and population variance; γ and β are learned. Epsilon is inside the square root. Ordinary inference uses stored running statistics. The linked full lesson calculates every stage for [2,4].

LaTeX: \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i

LaTeX: \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2

LaTeX: \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}

LaTeX: y_i=\gamma\hat{x}_i+\beta

Formula 47: Global average pooling

Take a separate average for each full spatial feature map. H and W count rows and columns; c selects one of C channels; a is an activation value. Output has C values. The linked lesson computes two channel averages and the following dense-layer parameters.

LaTeX: g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c}

Formula 48: Top-k error percentage

N is the total number of evaluated examples. C_k counts those whose true class is among the k highest-scoring predictions. Count failures, divide by total, then convert to percent. The worked lesson contrasts top-1 and top-5 using the same 100 images.

LaTeX: E_k=\frac{N-C_k}{N}\times100\%

Expanded formula guide and worked examples

Mean, variance and standard deviation

The mean is a measure of location; variance and standard deviation quantify dispersion. Distinguish population and sample conventions before calculating these statistics.

LaTeX: \mu=\frac{1}{n}\sum_i x_i

LaTeX: v=\frac{1}{n}\sum_i(x_i-\mu)^2

LaTeX: s=\sqrt{v}

Mean, variance and standard deviation from ordinary arithmetic

Definition. The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units.

LaTeX: \mu=\frac{1}{N}\sum_{i=1}^{N}x_i,\qquad \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2,\qquad \sigma=\sqrt{\sigma^2}

These are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable.

Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].

Given

Observations = [3,3,7,7]; N = 4.

Use population variance (divide by N).

To find

Find the population mean μ, variance σ² and standard deviation σ.

Formula — which one and why

Mean and spread. The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root.

N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms.

LaTeX: \mu=\frac{1}{N}\sum_{i=1}^{N}x_i

LaTeX: \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2

LaTeX: \sigma=\sqrt{\sigma^2}

Yaad rakhne ka cue: Mean = sabka total ÷ count; variance = squared distances ka mean.

Solution — step by step

Step 1: add all four observations.

LaTeX: 3+3+7+7=20

Step 2: divide by the count to obtain the mean.

LaTeX: \mu=\frac{20}{4}=5

Step 3: subtract the mean from each observation.

LaTeX: [3-5,3-5,7-5,7-5]=[-2,-2,2,2]

Step 4: square each deviation.

LaTeX: [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4]

Step 5: add the squared deviations.

LaTeX: 4+4+4+4=16

Step 6: divide by four for population variance.

LaTeX: \sigma^2=\frac{16}{4}=4

Step 7: take the nonnegative square root.

LaTeX: \sigma=\sqrt4=2

Answer: mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3.

Population mean and spread: [1, 2, 3]

Question. For the population [1, 2, 3], find mean, population variance and standard deviation.

Given

Values = [1, 2, 3]; N = 3. Use population division by N.

To find

Mean μ, variance σ² and standard deviation σ.

Formula — which one and why

The question specifies a population, so use N rather than the N−1 sample-variance divisor.

μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.

LaTeX: \mu=\frac1N\sum_i x_i

LaTeX: \sigma^2=\frac1N\sum_i(x_i-\mu)^2

LaTeX: \sigma=\sqrt{\sigma^2}

Yaad rakhne ka cue: Mean, distance, square, mean again, then root.

Solution — step by step

Step 1: add all values.

LaTeX: 1+2+3=6

Step 2: divide by the number of values.

LaTeX: \mu=\frac{6}{3}=2

Step 3: find and square deviation 1.

LaTeX: (1-2)=-1

LaTeX: (-1)^2=1

Step 4: find and square deviation 2.

LaTeX: (2-2)=0

LaTeX: (0)^2=0

Step 5: find and square deviation 3.

LaTeX: (3-2)=1

LaTeX: (1)^2=1

Step 6: average the squared deviations.

LaTeX: \sigma^2=\frac{1+0+1}{3}\approx0.666666667

Step 7: take the nonnegative square root.

LaTeX: \sigma=\sqrt{0.666666667}\approx0.816496581

Final answer: μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581.

Population mean and spread: [2, 2, 2]

Question. For the population [2, 2, 2], find mean, population variance and standard deviation.

Given

Values = [2, 2, 2]; N = 3. Use population division by N.

To find

Mean μ, variance σ² and standard deviation σ.

Formula — which one and why

The question specifies a population, so use N rather than the N−1 sample-variance divisor.

μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.

LaTeX: \mu=\frac1N\sum_i x_i

LaTeX: \sigma^2=\frac1N\sum_i(x_i-\mu)^2

LaTeX: \sigma=\sqrt{\sigma^2}

Yaad rakhne ka cue: Mean, distance, square, mean again, then root.

Solution — step by step

Step 1: add all values.

LaTeX: 2+2+2=6

Step 2: divide by the number of values.

LaTeX: \mu=\frac{6}{3}=2

Step 3: find and square deviation 1.

LaTeX: (2-2)=0

LaTeX: (0)^2=0

Step 4: find and square deviation 2.

LaTeX: (2-2)=0

LaTeX: (0)^2=0

Step 5: find and square deviation 3.

LaTeX: (2-2)=0

LaTeX: (0)^2=0

Step 6: average the squared deviations.

LaTeX: \sigma^2=\frac{0+0+0}{3}\approx0.000000000

Step 7: take the nonnegative square root.

LaTeX: \sigma=\sqrt{0.000000000}\approx0.000000000

Final answer: μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation.

Standardization

Standardization has a meaning before it has a formula: measure how far an input is from its training mean, then express that distance in standard-deviation units. The worked question below calculates the mean and standard deviation first, so neither number appears without explanation.

LaTeX: x_{scaled}=\frac{x-\mu_{train}}{s_{train}}

Standardization: what the answer 1 actually means

Definition. Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units.

LaTeX: z=\frac{x-\mu_{\mathrm{train}}}{\sigma_{\mathrm{train}}},\qquad \sigma_{\mathrm{train}}>0

Subtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1].

Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.

Given

Training observations = [3,3,7,7].

New observation = 9.

Estimate the population standard deviation from the training values.

To find

Find the training mean and standard deviation; standardize all four training values and the new observation.

Formula — which one and why

Mean and spread. The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root.

N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms.

LaTeX: \mu=\frac{1}{N}\sum_{i=1}^{N}x_i

LaTeX: \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2

LaTeX: \sigma=\sqrt{\sigma^2}

Yaad rakhne ka cue: Mean = sabka total ÷ count; variance = squared distances ka mean.

Standardization. Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units.

x = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value.

LaTeX: z=\frac{x-\mu}{\sigma}

Yaad rakhne ka cue: Pehle centre hatao, phir spread se divide karo.

Solution — step by step

Step 1: add all four observations.

LaTeX: 3+3+7+7=20

Step 2: divide by the count to obtain the mean.

LaTeX: \mu=\frac{20}{4}=5

Step 3: subtract the mean from each observation.

LaTeX: [3-5,3-5,7-5,7-5]=[-2,-2,2,2]

Step 4: square each deviation.

LaTeX: [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4]

Step 5: add the squared deviations.

LaTeX: 4+4+4+4=16

Step 6: divide by four for population variance.

LaTeX: \sigma^2=\frac{16}{4}=4

Step 7: take the nonnegative square root.

LaTeX: \sigma=\sqrt4=2

Step 8: standardize each original training observation.

LaTeX: \frac{3-5}{2}=-1

LaTeX: \frac{7-5}{2}=1

The repeated observations give [-1, −1, 1, 1].

Step 9: subtract the training mean from the new observation.

LaTeX: 9-5=4

Step 10: divide by the training standard deviation.

LaTeX: z_{\mathrm{new}}=\frac42=2

Answer: the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy.

Standardization with x = 3

Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.

Given

x = 3; training mean μ = 5; training standard deviation σ = 2.

To find

Standardized value z and its meaning.

Formula — which one and why

Subtract the training centre, then measure that distance in units of the training standard deviation.

A negative z means below the training mean; positive means above it; zero means equal to it.

LaTeX: z=\frac{x-\mu}{\sigma}

Yaad rakhne ka cue: Centre hatao, spread se divide; answer batata hai kitne σ door.

Solution — step by step

Step 1: find distance from the mean.

LaTeX: x-\mu=3-5=-2

Step 2: divide by one standard-deviation unit.

LaTeX: z=\frac{-2}{2}=-1

Step 3: interpret sign and size.

The value is 1 standard-deviation unit(s) below the training mean.

Final answer: z = -1. Use the same training statistics when transforming a new test input.

Standardization with x = 5

Question. Training mean is 5 and training standard deviation is 2. Standardize x = 5 and interpret the result.

Given

x = 5; training mean μ = 5; training standard deviation σ = 2.

To find

Standardized value z and its meaning.

Formula — which one and why

Subtract the training centre, then measure that distance in units of the training standard deviation.

A negative z means below the training mean; positive means above it; zero means equal to it.

LaTeX: z=\frac{x-\mu}{\sigma}

Yaad rakhne ka cue: Centre hatao, spread se divide; answer batata hai kitne σ door.

Solution — step by step

Step 1: find distance from the mean.

LaTeX: x-\mu=5-5=0

Step 2: divide by one standard-deviation unit.

LaTeX: z=\frac{0}{2}=0

Step 3: interpret sign and size.

The value is 0 standard-deviation unit(s) away from the training mean.

Final answer: z = 0. Use the same training statistics when transforming a new test input.

Standardization with x = 9

Question. Training mean is 5 and training standard deviation is 2. Standardize x = 9 and interpret the result.

Given

x = 9; training mean μ = 5; training standard deviation σ = 2.

To find

Standardized value z and its meaning.

Formula — which one and why

Subtract the training centre, then measure that distance in units of the training standard deviation.

A negative z means below the training mean; positive means above it; zero means equal to it.

LaTeX: z=\frac{x-\mu}{\sigma}

Yaad rakhne ka cue: Centre hatao, spread se divide; answer batata hai kitne σ door.

Solution — step by step

Step 1: find distance from the mean.

LaTeX: x-\mu=9-5=4

Step 2: divide by one standard-deviation unit.

LaTeX: z=\frac{4}{2}=2

Step 3: interpret sign and size.

The value is 2 standard-deviation unit(s) above the training mean.

Final answer: z = 2. Use the same training statistics when transforming a new test input.

Power, constant and sum derivatives

Derivatives describe local rates of change. Partial derivatives hold other independent variables fixed; their collection forms the gradient used for optimization.

LaTeX: \frac{d}{dx}(cx^n)=cnx^{n-1}

LaTeX: \frac{d}{dx}c=0

LaTeX: (f+g)\prime=f\prime+g\prime

Derivatives without a calculus course

Definition. A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector.

LaTeX: \nabla_\theta J=\begin{bmatrix}\partial J/\partial\theta_1\\\vdots\\\partial J/\partial\theta_d\end{bmatrix}

The rules needed for the supplied neural-network numericals are:

A parameter derivative is not an input derivative

For z = wx + b, the variable named in the denominator determines which derivative is required.

LaTeX: \frac{\partial z}{\partial w}=x,\qquad\frac{\partial z}{\partial x}=w,\qquad\frac{\partial z}{\partial b}=1

At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities.

Gradient direction

For a small parameter displacement Δθ, first-order Taylor expansion gives:

LaTeX: J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta

The gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large.

Differentiate before substituting: 3w^2+5

Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.

Given

J(w) = 3w^2+5; w = 2.

To find

The function value and its derivative value.

Formula — which one and why

The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.

c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.

LaTeX: \frac{d(cw^n+k)}{dw}=cnw^{n-1}

Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.

Solution — step by step

Step 1: evaluate the function itself.

LaTeX: J(2)=3(2)^2+5=17

Step 2: differentiate symbolically.

LaTeX: J\prime(w)=3(2)w^{2-1}+0

Step 3: put the given w into the derivative.

LaTeX: J\prime(2)=3(2)(2)^{1}=12

Final answer: J = 17; derivative = 12.

Differentiate before substituting: 0.7w^3+0

Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.

Given

J(w) = 0.7w^3+0; w = 1.

To find

The function value and its derivative value.

Formula — which one and why

The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.

c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.

LaTeX: \frac{d(cw^n+k)}{dw}=cnw^{n-1}

Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.

Solution — step by step

Step 1: evaluate the function itself.

LaTeX: J(1)=0.7(1)^3+0=0.7

Step 2: differentiate symbolically.

LaTeX: J\prime(w)=0.7(3)w^{3-1}+0

Step 3: put the given w into the derivative.

LaTeX: J\prime(1)=0.7(3)(1)^{2}=2.1

Final answer: J = 0.7; derivative = 2.1.

Differentiate before substituting: 2w^1+4

Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.

Given

J(w) = 2w^1+4; w = -1.

To find

The function value and its derivative value.

Formula — which one and why

The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.

c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.

LaTeX: \frac{d(cw^n+k)}{dw}=cnw^{n-1}

Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.

Solution — step by step

Step 1: evaluate the function itself.

LaTeX: J(-1)=2(-1)^1+4=2

Step 2: differentiate symbolically.

LaTeX: J\prime(w)=2(1)w^{1-1}+0

Step 3: put the given w into the derivative.

LaTeX: J\prime(-1)=2(1)(-1)^{0}=2

Final answer: J = 2; derivative = 2.

Chain rule

The chain rule differentiates a composite function. Multiply local derivatives along a dependency path and add the contributions of different paths to the same variable.

LaTeX: \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw}

The chain rule is multiplying linked effects

Definition. The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions.

First understand why the rates multiply

Suppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency.

The notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w.

Yaad rakhne ka cue: ek raaste par slopes multiply; alag raaston ke full answers add.

Chain-rule example 1: two linear links

Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.

Given

w = 1; u = 3w+1; v = 2u.

To find

The two function values and the rate dv/dw.

Formula — which one and why

v depends on w through u. Both links must be included, so multiply the two local derivatives.

du/dw is the slope of 3w+1. dv/du is the slope of 2u.

LaTeX: \frac{dv}{dw}=\frac{dv}{du}\frac{du}{dw}

Yaad rakhne ka cue: Andar ka slope × bahar ka slope.

Solution — step by step

Step 1: calculate the intermediate value.

LaTeX: u=3(1)+1=4

Step 2: calculate the final value.

LaTeX: v=2(4)=8

Step 3: differentiate the first link.

LaTeX: \frac{du}{dw}=3

Step 4: differentiate the second link.

LaTeX: \frac{dv}{du}=2

Step 5: multiply along the path.

LaTeX: \frac{dv}{dw}=2(3)=6

Final answer: u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different.

2. One output: calculate both parameter gradients

Question. A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1.

Given

Input x = 2; target y = 4.

Old weight w = 1; old bias b = 0.

Linear prediction z = wx+b; loss L = ½(z−y)².

Learning rate η = 0.1; update both parameters once.

To find

Find the loss gradient for w and for b, then w_new and b_new. Finally evaluate the changed prediction and loss.

Formula — which one and why

Chain rule for a linear neuron. The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.

z = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.

LaTeX: z=wx+b

LaTeX: L=\frac12(z-y)^2

LaTeX: g_w=\frac{\partial L}{\partial w}=(z-y)x

LaTeX: g_b=\frac{\partial L}{\partial b}=z-y

LaTeX: w_{\mathrm{new}}=w-\eta g_w

LaTeX: b_{\mathrm{new}}=b-\eta g_b

Yaad rakhne ka cue: Ek path mein multiply; alag paths mil rahe hon toh add.

Solution — step by step

Diagram: The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.. Labels: One output: values and derivatives | One output: values and derivatives | Given: x = 2, y = 4, w = 1, b = 0 | FORWARD VALUES | z = wx + b | z = 2 | e = z − y | e = −2 | L = ½e² | L = 2 | BACKWARD DERIVATIVES | Loss derivative reaches the neuron | ∂L/∂z = (∂L/∂e)(∂e/∂z) = (−2)(1) = −2 | Weight: multiply by x = 2 | ∂L/∂w = (−2)(2) = −4 | Bias: multiply by 1 | ∂L/∂b = (−2)(1) = −2 | Then update: w → 1.4 and b → 0.2 (η = 0.1).

Step 1 — evaluate the original model. Use the given parameters before applying any update.

LaTeX: z=wx+b=1(2)+0=2

LaTeX: e=z-y=2-4=-2

LaTeX: L=\frac12e^2=\frac12(-2)^2=2

The prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide.

Step 2 — differentiate the loss, keeping the target constant. First differentiate the square, then the error inside it.

LaTeX: \frac{\partial L}{\partial e}=\frac12(2e)=e

LaTeX: \frac{\partial e}{\partial z}=\frac{\partial(z-y)}{\partial z}=1

LaTeX: \delta_z=\frac{\partial L}{\partial z}=e(1)=-2

The symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection.

Step 3 — differentiate the neuron's affine expression. When differentiating with respect to w, x is a constant coefficient. When differentiating with respect to b, the coefficient of b is 1.

LaTeX: \frac{\partial z}{\partial w}=x=2

LaTeX: \frac{\partial z}{\partial b}=1

Step 4 — multiply along each parameter's path.

LaTeX: g_w=\frac{\partial L}{\partial w}=\delta_z\frac{\partial z}{\partial w}=(-2)(2)=-4

LaTeX: g_b=\frac{\partial L}{\partial b}=\delta_z\frac{\partial z}{\partial b}=(-2)(1)=-2

Both derivatives use the original forward-pass values. The weight and bias are different parameters, so keep their gradients separate.

Step 5 — update both parameters simultaneously. Gradient descent subtracts the learning rate times the corresponding gradient.

LaTeX: w_{\mathrm{new}}=w-\eta g_w=1-0.1(-4)=1.4

LaTeX: b_{\mathrm{new}}=b-\eta g_b=0-0.1(-2)=0.2

Subtracting a negative gradient increases the parameter. Do not recompute the bias gradient using the already changed weight; that would be a different update procedure.

Step 6 — perform a new forward pass to evaluate the update.

LaTeX: z_{\mathrm{new}}=1.4(2)+0.2=3

LaTeX: L_{\mathrm{new}}=\frac12(3-4)^2=0.5

Final answer: weight gradient −4, bias gradient −2; updated weight 1.4 and bias 0.2; new prediction 3 and loss 0.5. If a different question fixes the bias at zero, update only the weight; its new prediction would be 2.8, not 3.

3. Two outputs: why backward contributions are added

A shared intermediate value can influence more than one output. The derivative with respect to that shared value must include every output path. Adding paths happens before passing the accumulated derivative further backward.

Question. A shared linear hidden value is h = wx + b. Two linear outputs are o₁ = 2h and o₂ = −h; their targets are y₁ = 1 and y₂ = 0. The total loss is L = ½(o₁ − y₁)² + ½(o₂ − y₂)². At x = 2, w = 0.5 and b = 0.1, calculate the loss and the gradients with respect to h, w and b. Treat the output multipliers 2 and −1 as fixed.

Given

h = wx+b; outputs o₁ = 2h and o₂ = −h.

x = 2; w = 0.5; b = 0.1; targets = [1,0].

Loss = half the sum of the two squared output errors.

Output multipliers 2 and −1 stay fixed.

To find

Find total loss and its derivatives with respect to h, w and b.

Formula — which one and why

Chain rule with more than one path. The shared hidden value affects each output. Multiply derivatives within each path, then add the complete path contributions before continuing to w and b.

oⱼ = output j; yⱼ = its target; h = shared hidden value; ∂oⱼ/∂h is the multiplier on that output path.

LaTeX: L=\frac12\sum_j(o_j-y_j)^2

LaTeX: \frac{\partial L}{\partial h}=\sum_j(o_j-y_j)\frac{\partial o_j}{\partial h}

LaTeX: \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x

LaTeX: \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h}

Yaad rakhne ka cue: Har raaste ka effect pehle multiply, junction par sab add.

Solution — step by step

Diagram: The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.. Labels: Two outputs: add complete path contributions | Two outputs: add complete path contributions | h = wx + b = 0.5(2) + 0.1 = 1.1 | Shared hidden value | h = 1.1 | o₁ = 2h; target y₁ = 1 | o₁ = 2.2 | o₂ = −h; target y₂ = 0 | o₂ = −1.1 | × 2 | × (−1) | Output derivative: 2.2 − 1 = 1.2 | Output derivative: −1.1 − 0 = −1.1 | BACK TO h | Path 1: (1.2)(2) = 2.4 | Path 2: (−1.1)(−1) = 1.1 | Add at the shared value, then continue backward | ∂L/∂h = 2.4 + 1.1 = 3.5 | ∂L/∂w = 3.5 × 2 = 7 | ∂L/∂b = 3.5 × 1 = 3.5

Step 1 — forward values.

LaTeX: h=0.5(2)+0.1=1.1

LaTeX: o_1=2(1.1)=2.2,\qquad o_2=-(1.1)=-1.1

LaTeX: L_1=\frac12(2.2-1)^2=0.72

LaTeX: L_2=\frac12(-1.1-0)^2=0.605

LaTeX: L=L_1+L_2=0.72+0.605=1.325

Step 2 — output derivatives. Each half squared loss differentiates to prediction minus target. The other loss does not depend on this output.

LaTeX: \frac{\partial L}{\partial o_1}=o_1-y_1=2.2-1=1.2

LaTeX: \frac{\partial L}{\partial o_2}=o_2-y_2=-1.1-0=-1.1

Step 3 — calculate the contribution of each complete path. The derivative of 2h with respect to h is 2; the derivative of −h is −1.

LaTeX: \frac{\partial L_1}{\partial h}=(1.2)(2)=2.4

LaTeX: \frac{\partial L_2}{\partial h}=(-1.1)(-1)=1.1

Step 4 — add the two contributions at the shared hidden value.

LaTeX: \frac{\partial L}{\partial h}=\frac{\partial L_1}{\partial h}+\frac{\partial L_2}{\partial h}

LaTeX: \frac{\partial L}{\partial h}=2.4+1.1=3.5

Adding the raw output errors, 1.2 + (−1.1), would omit the output multipliers. Each error must first be multiplied by the local derivative on its own path.

Step 5 — continue to the hidden weight and bias. Since h = wx + b, its derivatives with respect to w and b are x and 1.

LaTeX: \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x=3.5(2)=7

LaTeX: \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h}(1)=3.5

Final answer: L = 1.325, ∂L/∂h = 3.5, ∂L/∂w = 7 and ∂L/∂b = 3.5. If the hidden value instead includes a nonlinear activation, multiply the accumulated downstream derivative by that activation's derivative before multiplying by x. This is the hidden-delta calculation in multilayer backpropagation.

Chain-rule example 4: a square after a linear link

Question. Let z = 3w+1 and L = z². At w = 1, find z, L and dL/dw.

Given

w = 1; z = 3w+1; L = z².

To find

z, L and the derivative of L with respect to w.

Formula — which one and why

The square acts on z, not directly on w. Differentiate the square first and include the change of z with w.

dL/dz = slope of the outer square; dz/dw = slope of the inner linear expression.

LaTeX: \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw}

LaTeX: \frac{d(z^2)}{dz}=2z

Yaad rakhne ka cue: Square ka 2z; andar ka 3; dono multiply.

Solution — step by step

Step 1: evaluate the inner expression.

LaTeX: z=3(1)+1=4

Step 2: evaluate the loss expression.

LaTeX: L=4^2=16

Step 3: differentiate the square and use z = 4.

LaTeX: \frac{dL}{dz}=2z=2(4)=8

Step 4: differentiate the inner expression.

LaTeX: \frac{dz}{dw}=3

Step 5: multiply, because the changes occur in sequence.

LaTeX: \frac{dL}{dw}=8(3)=24

Final answer: z = 4, L = 16 and dL/dw = 24. The combined operation is multiplication, not 8+3.

Drill 6 Chain rule on a tiny loss

Answer — step by step

Given

Model: o = wx+b; loss L = ½(o−y)².

x = 3; y = 1; w = 0.2; b = 0.1.

Learning rate η = 0.1.

To find

Find ∂L/∂w, ∂L/∂b and the updated weight and bias.

Formula — which one and why

Chain rule for a linear neuron. The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.

z = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.

LaTeX: z=wx+b

LaTeX: L=\frac12(z-y)^2

LaTeX: g_w=\frac{\partial L}{\partial w}=(z-y)x

LaTeX: g_b=\frac{\partial L}{\partial b}=z-y

LaTeX: w_{\mathrm{new}}=w-\eta g_w

LaTeX: b_{\mathrm{new}}=b-\eta g_b

Yaad rakhne ka cue: Ek path mein multiply; alag paths mil rahe hon toh add.

Solution — step by step

Meaning and formulas

The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects.

LaTeX: o=wx+b,\qquad L=\frac12(o-y)^2

LaTeX: \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1

LaTeX: \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y

The learning rate η says how much of the gradient to subtract.

LaTeX: w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b}

Step 1: obtain the current prediction and error

LaTeX: o=(0.2\times3)+0.1=0.6+0.1=0.7

LaTeX: o-y=0.7-1=-0.3

LaTeX: L=\frac12(-0.3)^2=\frac12(0.09)=0.045

Step 2: calculate both gradients from this same prediction

LaTeX: \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9

LaTeX: \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3

Step 3: update the weight

LaTeX: w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29

Step 4: update the bias

LaTeX: b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13

Final answer: weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass.

Dot product and a neuron

A neuron starts with matched input–weight pairs. Multiply inside each pair, add the products, add the bias once, and then apply the activation. The summation sign is simply a shorter way to write those additions.

LaTeX: z=\sum_i w_i x_i+b

LaTeX: a=f(z)

Model answer: explain an artificial neuron

Answer — step by step

An artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers.

Diagram: Follow each input through its weight, add one bias at the receiving neuron, then apply activation.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate

Formula and symbols

LaTeX: z=w_1x_1+w_2x_2+\cdots+w_nx_n+b,\qquad a=f(z)

Part	Meaning	What it does
x₁, x₂, …	Input features	Supply the information for this calculation
w₁, w₂, …	Weights	Control the contribution of each matching input
b	Bias	Shifts the weighted sum independently of the inputs
z	Net input	The number before activation
f	Activation rule	Transforms z, often nonlinearly
a	Activated output	The number sent onward

Worked ReLU neuron example

Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.

Given

Inputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU.

To find

Preactivation z and activated output a.

Formula — which one and why

Each input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total.

z means before activation; a means after activation.

LaTeX: z=w_1x_1+w_2x_2+b

LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Product, product, sum, bias, phir activation.

Solution — step by step

Step 1: multiply the first pair.

LaTeX: 0.5(2)=1

Step 2: multiply the second pair.

LaTeX: (-0.2)(1)=-0.2

Step 3: add both contributions.

LaTeX: 1+(-0.2)=0.8

Step 4: add the bias once.

LaTeX: z=0.8+0.1=0.9

Step 5: apply ReLU.

LaTeX: a=\max(0,0.9)=0.9

Final answer: z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged.

The weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once.

What one artificial neuron does

Definition. An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output.

What goes into a neuron?

An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive.

The bias is one extra adjustable number added to the total. The total before activation is called z or preactivation. The activation function is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the activation.

Yaad rakhne ka cue: inputs × apne weights → products ka sum → ek bias → activation → output.

Mathematical model and notation

LaTeX: z=\sum_{i=1}^{d}w_ix_i+b=w^Tx+b

LaTeX: a=f(z)

The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs.

Why both a bias and an activation are needed

The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin.

The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values.

Diagram: Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate

Worked numerical

Question. A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output.

Given

Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5].

Bias = 0.1; activation = sigmoid.

To find

Find the preactivation z, then the sigmoid output.

Formula — which one and why

Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.

LaTeX: z=\sum_iw_ix_i+b

LaTeX: a=f(z)

Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.

Sigmoid output. The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.

z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.

LaTeX: a=\frac1{1+e^{-z}}

Yaad rakhne ka cue: Minus-z ki e power, plus one, phir one divided by whole denominator.

Solution — step by step

Step 1: multiply input 1 by weight 1.

LaTeX: x_1w_1=2(0.4)=0.8

Step 2: multiply input 2 by weight 2.

LaTeX: x_2w_2=1(-0.2)=-0.2

The minus sign belongs to the weight, so the contribution stays negative.

Step 3: multiply input 3 by weight 3.

LaTeX: x_3w_3=3(0.5)=1.5

Step 4: add the three contributions.

LaTeX: s=0.8+(-0.2)+1.5=0.6+1.5=2.1

Step 5: add the bias once.

LaTeX: z=s+b=2.1+0.1=2.2

This is the preactivation. It is not yet the requested sigmoid output.

Step 6: negate z for the exponential.

LaTeX: -z=-2.2

LaTeX: e^{-z}=e^{-2.2}\approx0.110803158

Step 7: calculate the complete denominator.

LaTeX: 1+e^{-z}=1+0.110803158=1.110803158

Step 8: divide one by that denominator.

LaTeX: a=\frac1{1+e^{-z}}=\frac1{1.110803158}\approx0.900249511

Final answer: sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025.

From one neuron to a layer

With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron:

LaTeX: z=Wx+b,\qquad a=f(z)

Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together.

Neuron example 2: ReLU output

Question. A neuron receives inputs [2], weights [3] and bias 1. Calculate its preactivation and ReLU output.

Given

Inputs = [2].

Weights = [3]; bias = 1.

Activation = ReLU.

To find

z before activation and a after activation.

Formula — which one and why

The weighted sum combines the inputs. The output must then use the activation explicitly requested.

z names the sum; a names the activated result. They are not always equal.

LaTeX: z=\sum_i w_ix_i+b

LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.

Solution — step by step

Step 1: multiply input 1 by its own weight.

LaTeX: (2)(3)=6

Step 2: add the input contributions.

LaTeX: s=(6)=6

Step 3: add the bias once.

LaTeX: z=6+(1)=7

Step 4: apply the stated activation.

LaTeX: a=\max(0,7)=7

Final answer: z = 7; a = 7.

Neuron example 3: ReLU output

Question. A neuron receives inputs [2, 1], weights [-1, 0.5] and bias 0.2. Calculate its preactivation and ReLU output.

Given

Inputs = [2, 1].

Weights = [-1, 0.5]; bias = 0.2.

Activation = ReLU.

To find

z before activation and a after activation.

Formula — which one and why

The weighted sum combines the inputs. The output must then use the activation explicitly requested.

z names the sum; a names the activated result. They are not always equal.

LaTeX: z=\sum_i w_ix_i+b

LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.

Solution — step by step

Step 1: multiply input 1 by its own weight.

LaTeX: (2)(-1)=-2

Step 2: multiply input 2 by its own weight.

LaTeX: (1)(0.5)=0.5

Step 3: add the input contributions.

LaTeX: s=(-2)+(0.5)=-1.5

Step 4: add the bias once.

LaTeX: z=-1.5+(0.2)=-1.3

Step 5: apply the stated activation.

LaTeX: a=\max(0,-1.3)=0

Final answer: z = -1.3; a = 0.

Neuron example 4: sigmoid output

Question. A neuron receives inputs [1, 2], weights [0.2, -0.1] and bias 0. Calculate its preactivation and sigmoid output.

Given

Inputs = [1, 2].

Weights = [0.2, -0.1]; bias = 0.

Activation = sigmoid.

To find

z before activation and a after activation.

Formula — which one and why

The weighted sum combines the inputs. The output must then use the activation explicitly requested.

z names the sum; a names the activated result. They are not always equal.

LaTeX: z=\sum_i w_ix_i+b

LaTeX: a=\frac1{1+e^{-z}}

Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.

Solution — step by step

Step 1: multiply input 1 by its own weight.

LaTeX: (1)(0.2)=0.2

Step 2: multiply input 2 by its own weight.

LaTeX: (2)(-0.1)=-0.2

Step 3: add the input contributions.

LaTeX: s=(0.2)+(-0.2)=0

Step 4: add the bias once.

LaTeX: z=0+(0)=0

Step 5: apply the stated activation.

LaTeX: a=\frac1{1+e^{0}}=\frac1{1+1}=0.5

Final answer: z = 0; a = 0.5.

Drill 4 Dot product and bias

Answer — step by step

Given

Inputs x = [2, −1, 0.5].

Weights w = [0.3, 0.4, −0.2]; bias b = 0.1.

Activation: ReLU.

To find

Find the weighted sum z and ReLU output.

Formula — which one and why

Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.

LaTeX: z=\sum_iw_ix_i+b

LaTeX: a=f(z)

Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

LaTeX: \operatorname{ReLU}(z)=\max(0,z)

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Solution — step by step

Meaning and formulas

A dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero.

LaTeX: z=w_1x_1+w_2x_2+w_3x_3+b,\qquad a=\max(0,z)

Here x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output.

Step 1: multiply the matching pairs

LaTeX: w_1x_1=0.3\times2=0.6

LaTeX: w_2x_2=0.4\times(-1)=-0.4

LaTeX: w_3x_3=(-0.2)\times0.5=-0.1

Step 2: add all products and add bias once

LaTeX: z=0.6+(-0.4)+(-0.1)+0.1

LaTeX: z=0.2-0.1+0.1=0.2

Step 3: apply the activation

LaTeX: a=\max(0,0.2)=0.2

Final answer: weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive.

Matrix multiplication

Matrix multiplication implements affine transformations in a dense layer. Dimensions determine which products are valid; each output component is the dot product of a weight row and the input vector, plus its bias.

LaTeX: (AB)_{ij}=\sum_k A_{ik}B_{kj}

Matrix multiplication: every row has a job

Row, column, vector and matrix — read these first

A row runs horizontally, from left to right. A column runs vertically, from top to bottom. A matrix is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows.

A vector is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors.

Transpose: rows become columns

The superscript T means transpose, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m.

LaTeX: [A^T]_{ij}=A_{ji}

The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1.

Yaad rakhne ka cue: transpose = row ko column bana; number aur sign ko haath mat laga.

Transpose example 1

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

LaTeX: A=\begin{bmatrix}2&1&3\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 1 row(s) and 3 column(s), so its size is 1 × 3.

Step 2: move original row 1 into new column 1.

Copy [2, 1, 3] down column 1, in the same order.

Step 3: write the transposed matrix.

LaTeX: A^T=\begin{bmatrix}2\\1\\3\end{bmatrix}

Step 4: state its new size.

The result has 3 row(s) and 1 column(s): 3 × 1.

Final answer: Aᵀ has size 3 × 1. Transposing it again gives the original A.

Transpose example 2

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

LaTeX: A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 2 row(s) and 3 column(s), so its size is 2 × 3.

Step 2: move original row 1 into new column 1.

Copy [1, 2, 3] down column 1, in the same order.

Step 3: move original row 2 into new column 2.

Copy [4, 5, 6] down column 2, in the same order.

Step 4: write the transposed matrix.

LaTeX: A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix}

Step 5: state its new size.

The result has 3 row(s) and 2 column(s): 3 × 2.

Final answer: Aᵀ has size 3 × 2. Transposing it again gives the original A.

Transpose example 3

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

LaTeX: A=\begin{bmatrix}1&-2\\0&3\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 2 row(s) and 2 column(s), so its size is 2 × 2.

Step 2: move original row 1 into new column 1.

Copy [1, -2] down column 1, in the same order.

Step 3: move original row 2 into new column 2.

Copy [0, 3] down column 2, in the same order.

Step 4: write the transposed matrix.

LaTeX: A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix}

Step 5: state its new size.

The result has 2 row(s) and 2 column(s): 2 × 2.

Final answer: Aᵀ has size 2 × 2. Transposing it again gives the original A.

Transpose example 4

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

LaTeX: A=\begin{bmatrix}-1\\4\\2\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 3 row(s) and 1 column(s), so its size is 3 × 1.

Step 2: move original row 1 into new column 1.

Copy [-1] down column 1, in the same order.

Step 3: move original row 2 into new column 2.

Copy [4] down column 2, in the same order.

Step 4: move original row 3 into new column 3.

Copy [2] down column 3, in the same order.

Step 5: write the transposed matrix.

LaTeX: A^T=\begin{bmatrix}-1&4&2\end{bmatrix}

Step 6: state its new size.

The result has 1 row(s) and 3 column(s): 1 × 3.

Final answer: Aᵀ has size 1 × 3. Transposing it again gives the original A.

Dot product: multiply matching entries, then add

A dot product turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed.

LaTeX: w^Tx=w_1x_1+w_2x_2+w_3x_3

This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products.

Dot-product example 1

Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.

Given

Weights w = [2, 3].

Matching inputs x = [4, 1].

Both lists contain 2 numbers.

To find

One scalar value wᵀx.

Formula — which one and why

A dot product is a sum of matching products. There is no bias in this question.

Σ means add; the index i visits each matching pair.

LaTeX: w^Tx=\sum_i w_ix_i

Yaad rakhne ka cue: Same position multiply; sab products add.

Solution — step by step

Step 1: multiply matching pair 1.

LaTeX: (2)(4)=8

Step 2: multiply matching pair 2.

LaTeX: (3)(1)=3

Step 3: add the products.

LaTeX: w^Tx=(8)+(3)=11

Final answer: wᵀx = 11.

Dot-product example 2

Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.

Given

Weights w = [1, -2].

Matching inputs x = [2, 3].

Both lists contain 2 numbers.

To find

One scalar value wᵀx.

Formula — which one and why

A dot product is a sum of matching products. There is no bias in this question.

Σ means add; the index i visits each matching pair.

LaTeX: w^Tx=\sum_i w_ix_i

Yaad rakhne ka cue: Same position multiply; sab products add.

Solution — step by step

Step 1: multiply matching pair 1.

LaTeX: (1)(2)=2

Step 2: multiply matching pair 2.

LaTeX: (-2)(3)=-6

Step 3: add the products.

LaTeX: w^Tx=(2)+(-6)=-4

Final answer: wᵀx = -4.

Dot-product example 3

Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.

Given

Weights w = [0.5, 1, -1].

Matching inputs x = [4, 2, 3].

Both lists contain 3 numbers.

To find

One scalar value wᵀx.

Formula — which one and why

A dot product is a sum of matching products. There is no bias in this question.

Σ means add; the index i visits each matching pair.

LaTeX: w^Tx=\sum_i w_ix_i

Yaad rakhne ka cue: Same position multiply; sab products add.

Solution — step by step

Step 1: multiply matching pair 1.

LaTeX: (0.5)(4)=2

Step 2: multiply matching pair 2.

LaTeX: (1)(2)=2

Step 3: multiply matching pair 3.

LaTeX: (-1)(3)=-3

Step 4: add the products.

LaTeX: w^Tx=(2)+(2)+(-3)=1

Final answer: wᵀx = 1.

Matrix multiplication: each row makes one output

To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B.

LaTeX: C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj}

For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once.

Yaad rakhne ka cue: multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column.

Definition. A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation.

LaTeX: z=Wx+b

For d input features and m receiving neurons, the dimensions are:

LaTeX: W\in\mathbb R^{m\times d},\quad x\in\mathbb R^{d\times1},\quad b,z\in\mathbb R^{m\times1}

The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product.

Diagram: Each row of the weight matrix calculates one receiving neuron's preactivation.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result.

Question. Given W = LaTeX: \begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}, x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b.

Given

Weights W = LaTeX: \begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}.

Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ.

To find

Find both components of z = Wx+b.

Formula — which one and why

Row–column multiplication. Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.

Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.

LaTeX: z_j=\sum_i W_{ji}x_i+b_j

Yaad rakhne ka cue: Ek row × poora input column = ek receiving neuron ka answer.

Solution — step by step

Step 1: verify compatible dimensions.

W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column.

Step 2: calculate the first receiving row products.

LaTeX: 1(2)=2

LaTeX: (-2)(1)=-2

Step 3: sum row 1 and add its bias.

LaTeX: z_1=2+(-2)+0.1=0.1

Step 4: calculate the second receiving row products.

LaTeX: 0.5(2)=1

LaTeX: 3(1)=3

Step 5: sum row 2 and add its bias.

LaTeX: z_2=1+3-0.2=3.8

LaTeX: z_1=1(2)+(-2)(1)+0.1=0.1

LaTeX: z_2=0.5(2)+3(1)-0.2=3.8

LaTeX: z=\begin{bmatrix}0.1\\3.8\end{bmatrix}

An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W.

Two output columns

Question. Multiply A = LaTeX: \begin{bmatrix}1 & 2\\3 & 4\end{bmatrix} by B = LaTeX: \begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}. Show all four output entries.

Given

LaTeX: A=\begin{bmatrix}1&2\\3&4\end{bmatrix}

LaTeX: B=\begin{bmatrix}2&0\\1&3\end{bmatrix}

To find

The 2 × 2 matrix C = AB.

Formula — which one and why

Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication.

Cᵢⱼ is the entry in output row i, column j.

LaTeX: C_{ij}=\sum_k A_{ik}B_{kj}

Yaad rakhne ka cue: Output ki position dekho: wahi row, wahi column.

Solution — step by step

Step 1: check the dimensions.

(2 × 2)(2 × 2) is valid; output size = 2 × 2.

Step 2: row 1 with column 1.

LaTeX: C_{11}=1(2)+2(1)=2+2=4

Step 3: row 1 with column 2.

LaTeX: C_{12}=1(0)+2(3)=0+6=6

Step 4: row 2 with column 1.

LaTeX: C_{21}=3(2)+4(1)=6+4=10

Step 5: row 2 with column 2.

LaTeX: C_{22}=3(0)+4(3)=0+12=12

Step 6: assemble the output by position.

LaTeX: C=\begin{bmatrix}4&6\\10&12\end{bmatrix}

Final answer: C = LaTeX: \begin{bmatrix}4 & 6\\10 & 12\end{bmatrix}, with size 2 × 2.

Two products that must not be confused

Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.

Given

a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5.

To find

The 2 × 1 entry-by-entry product and the 2 × 2 outer product.

Formula — which one and why

⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons.

⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row.

LaTeX: (a\odot b)_i=a_i b_i

LaTeX: (ab^T)_{ij}=a_i b_j

Yaad rakhne ka cue: ⊙ mein same jagah; outer product mein har pair.

Solution — step by step

Step 1: multiply matching entries.

LaTeX: a\odot b=\begin{bmatrix}8\\15\end{bmatrix}

Step 2: transpose b.

LaTeX: b^T=\begin{bmatrix}4&5\end{bmatrix}

Step 3: form row 1 of the outer product.

LaTeX: 2(4)=8,\qquad2(5)=10

Step 4: form row 2 of the outer product.

LaTeX: 3(4)=12,\qquad3(5)=15

Step 5: write both rows.

LaTeX: ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix}

Final answer: Entry-by-entry result = [8,15]ᵀ. Outer product = LaTeX: \begin{bmatrix}8 & 10\\12 & 15\end{bmatrix}.

Drill 5 Matrix rows receive inputs

Question

Question. For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU.

Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W | 1 | 2 | −1 | 3

Answer — step by step

Given

W = LaTeX: \begin{bmatrix}1 & 2\\-1 & 3\end{bmatrix}.

Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ.

Activation: ReLU on each result.

To find

Find Wx+b and the activated output.

Formula — which one and why

Row–column multiplication. Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.

Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.

LaTeX: z_j=\sum_i W_{ji}x_i+b_j

Yaad rakhne ka cue: Ek row × poora input column = ek receiving neuron ka answer.

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

LaTeX: \operatorname{ReLU}(z)=\max(0,z)

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Solution — step by step

Meaning and formula

Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row.

LaTeX: z=Wx+b,\qquad a_j=\max(0,z_j)

LaTeX: W=\begin{bmatrix}1&2\\-1&3\end{bmatrix},\quad x=\begin{bmatrix}2\\1\end{bmatrix},\quad b=\begin{bmatrix}0.5\\-0.5\end{bmatrix}

Step 1: calculate receiving neuron 1 from row 1

LaTeX: z_1=(1\times2)+(2\times1)+0.5

LaTeX: z_1=2+2+0.5=4.5

Step 2: calculate receiving neuron 2 from row 2

LaTeX: z_2=((-1)\times2)+(3\times1)+(-0.5)

LaTeX: z_2=-2+3-0.5=0.5

Step 3: apply ReLU to each entry

LaTeX: a_1=\max(0,4.5)=4.5,\qquad a_2=\max(0,0.5)=0.5

LaTeX: z=\begin{bmatrix}4.5\\0.5\end{bmatrix},\qquad a=\begin{bmatrix}4.5\\0.5\end{bmatrix}

The shape is 2 rows by 1 column: one output for each receiving row.

Layer forward pass

A layer does the neuron calculation for several receiving neurons. Finish all its outputs before using them as the next layer’s inputs. The bracketed superscript is a layer label, not a power.

LaTeX: z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}

LaTeX: a^{(l)}=f(z^{(l)})

Forward propagation through multiple neurons

Definitions. A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations.

Input, hidden and output layers

The input layer supplies the feature vector. A hidden layer learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The output layer produces the prediction in the form needed by the task.

In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections.

Diagram: A feedforward network calculates each hidden representation before using it to calculate the next layer.. Labels: INPUT | HIDDEN | OUTPUT | x1 | x2 | h1 | h2 | o | Each hidden neuron receives both inputs and one bias. | Forward uses activations. Backward uses OLD weights.

General layer equations and dimensions

Let a⁽⁰⁾ = x. For layer l:

LaTeX: z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}

LaTeX: a^{(l)}=f^{(l)}(z^{(l)})

If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j.

LaTeX: W^{(l)}\in\mathbb R^{m\times n},\qquad b^{(l)},z^{(l)},a^{(l)}\in\mathbb R^{m\times1}

For one hidden layer, these computations compose to give:

LaTeX: h=f^{(1)}(W^{(1)}x+b^{(1)})

LaTeX: \hat y=f^{(2)}(W^{(2)}h+b^{(2)})

The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer.

Forward propagation versus training

Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture.

Complete two-layer numerical

Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = LaTeX: \begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix} and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.

Given

A feedforward network receives x = [1, 0]ᵀ.

Its hidden-layer weights are W¹ = LaTeX: \begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix} and biases are b¹ = [0.1, 0.2]ᵀ.

The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1.

Use ReLU in both layers and one row per receiving neuron.

To find

Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

LaTeX: z_h=W_1x+b_1

LaTeX: h=f_1(z_h)

LaTeX: z_o=W_2h+b_2

LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

LaTeX: \operatorname{ReLU}(z)=\max(0,z)

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Solution — step by step

Diagram: The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7

Step 1: identify the matrix dimensions. W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output.

Step 2: calculate the first hidden preactivation. Multiply the first row by x, then add the first hidden bias.

LaTeX: z^{(1)}_1=0.2(1)+0.4(0)+0.1=0.3

Step 3: calculate the second hidden preactivation.

LaTeX: z^{(1)}_2=0.3(1)+0.7(0)+0.2=0.5

Step 4: activate the hidden layer. Both values are positive, so ReLU retains them.

LaTeX: a^{(1)}_1=\max(0,0.3)=0.3

LaTeX: a^{(1)}_2=\max(0,0.5)=0.5

LaTeX: a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix}

Step 5: calculate the output preactivation. Use the activated hidden values as this layer's inputs.

LaTeX: z^{(2)}=0.5(0.3)+0.6(0.5)+0.1

LaTeX: z^{(2)}=0.15+0.30+0.10=0.55

Step 6: apply the output activation.

LaTeX: \hat y=\max(0,0.55)=0.55

Final answer: hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass.

Model answer: forward propagation, backpropagation and updating

Answer — step by step

Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs.

Diagram: Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.. Labels: The four operations in one training step | The four operations in one training step | Repeat this sequence using the current parameters. | Predict | Forward pass | Measure error | Compute loss | Find | gradients | Backpropagation | Adjust | Optimizer | update | Calculate all gradients before replacing the old weights.

Operation	Direction or action	Result
Forward propagation	Inputs move through successive layers	Predictions and intermediate activations
Loss calculation	Compare predictions with targets	A scalar objective for the example or batch
Backpropagation	Apply the chain rule from the loss toward earlier layers	Gradients of the objective with respect to parameters
Optimizer update	Use those gradients to change parameters	A new set of weights and biases

1. Forward propagation

At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next.

LaTeX: z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)})

2. Loss

Use the loss specified by the task. For one regression output, half squared loss is one possible choice:

LaTeX: L=\frac12(a-y)^2

Here a is the prediction and y is the target. Other stated losses produce different derivatives.

3. Backpropagation

Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation.

LaTeX: \frac{\partial L}{\partial w_{ji}}=\delta_j a_i

Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it.

4. Update and repeat

LaTeX: w_{\mathrm{new}}=w_{\mathrm{old}}-\eta\frac{\partial L}{\partial w}

Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations.

Backpropagation with one hidden neuron

Definition. Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters.

Network equations and the meaning of delta

Consider one input, one hidden neuron and one output neuron:

LaTeX: z_h=w_1x+b_1,\qquad h=f(z_h)

LaTeX: z_o=w_2h+b_2,\qquad\hat y=g(z_o)

For the half squared loss used in this example:

LaTeX: L=\frac12(\hat y-y)^2

Define a neuron's delta as the derivative of the loss with respect to its preactivation, not with respect to its output:

LaTeX: \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o)

The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂:

LaTeX: \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o

To reach the hidden neuron, continue backwards through the old output weight and the hidden activation:

LaTeX: \delta_h=\delta_o w_2 f'(z_h)

LaTeX: \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h

The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass.

Complete worked numerical

Question. x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1.

Given

x = 2, y = 1.

w₁ = 0.5, b₁ = 0.1.

w₂ = 0.8, b₂ = 0.2.

Both neurons use ReLU.

Use half squared loss.

Find the output and all four parameter gradients, then update all four parameters using η = 0.1.

To find

Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

LaTeX: z_h=W_1x+b_1

LaTeX: h=f_1(z_h)

LaTeX: z_o=W_2h+b_2

LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.

LaTeX: L=\frac12\sum_j(o_j-y_j)^2

LaTeX: \frac{\partial L}{\partial o_j}=o_j-y_j

Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection.

Step 1: forward propagation through the hidden neuron.

LaTeX: z_h=0.5(2)+0.1=1.1

LaTeX: h=\operatorname{ReLU}(1.1)=1.1

Step 2: forward propagation through the output neuron.

LaTeX: z_o=0.8(1.1)+0.2=1.08

LaTeX: \hat y=\operatorname{ReLU}(1.08)=1.08

ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here.

Step 3: calculate the loss and output delta.

LaTeX: L=\frac12(1.08-1)^2=0.0032

Both preactivations are positive, so both ReLU derivatives are 1.

LaTeX: \delta_o=(1.08-1)(1)=0.08

Step 4: calculate the output-layer parameter gradients.

LaTeX: \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088

LaTeX: \frac{\partial L}{\partial b_2}=0.08

Step 5: propagate the derivative to the hidden neuron.

Use w₂ = 0.8, its value during the forward pass:

LaTeX: \delta_h=0.08(0.8)(1)=0.064

LaTeX: \frac{\partial L}{\partial w_1}=0.064(2)=0.128

LaTeX: \frac{\partial L}{\partial b_1}=0.064

Step 6: update each parameter using its own gradient.

LaTeX: w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872

LaTeX: b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936

LaTeX: w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912

LaTeX: b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192

Final answer: the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together.

Drill 11 Full ReLU backpropagation

Question

Question. Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases.

Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5

Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass.

Answer — step by step

Given

Input x = [1, 2].

W₁ is printed with the question.

b₁ = [0.1, 0.2].

W₂ is printed with the question.

b₂ = [0.1].

Hidden and output activations are ReLU.

Target y = [1].

Learning rate η = 0.1.

Use half squared loss and update all weights and biases.

To find

Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

LaTeX: z_h=W_1x+b_1

LaTeX: h=f_1(z_h)

LaTeX: z_o=W_2h+b_2

LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.

LaTeX: L=\frac12\sum_j(o_j-y_j)^2

LaTeX: \frac{\partial L}{\partial o_j}=o_j-y_j

Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Formulas used before substituting numbers

For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.

LaTeX: z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2

Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.

LaTeX: \delta_{o_k}=(o_k-y_k)f'(z_{o_k})

LaTeX: \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j})

The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.

LaTeX: \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j

LaTeX: w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j

For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.

Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.

Hidden neuron 1:

LaTeX: z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5

LaTeX: h_{1} = \max(0,0.5) = 0.5

Hidden neuron 2:

LaTeX: z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3

LaTeX: h_{2} = \max(0,0.3) = 0.3

Output neuron 1:

LaTeX: z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45

LaTeX: o_{1} = \max(0,0.45) = 0.45

Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.

Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide.

Output delta 1:

LaTeX: \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55

The ReLU derivative is 1 here because this preactivation is positive.

Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.

h1 → o1:

Sending activation0.5h1

×

Receiving delta−0.55δ at o1

=

Connection gradient−0.275h1 → o1

Now update this weight

Old weight0.4

→

New weight0.4275

Subtract η × gradient = −0.0275. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

LaTeX: g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1

LaTeX: g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275

h2 → o1:

Sending activation0.3h2

×

Receiving delta−0.55δ at o1

=

Connection gradient−0.165h2 → o1

Now update this weight

Old weight0.5

→

New weight0.5165

Subtract η × gradient = −0.0165. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

LaTeX: g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2

LaTeX: g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165

Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.

Hidden 1 downstream contribution:

Calculate each outgoing path using the old outgoing weight and that output's delta.

Output 1

Output delta−0.55

×

Old outgoing weight0.4

=

Path contribution−0.22

↓

Add every path contribution−0.22Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

Contribution from output 1.

LaTeX: C_1=(0.4)\times(-0.55)\approx -0.22

Accumulate the contributions, then apply the hidden activation derivative below.

LaTeX: \text{Sum}=C_1\approx -0.22

LaTeX: \delta_{h_1} = -0.22 \times 1 = -0.22

Hidden 2 downstream contribution:

Calculate each outgoing path using the old outgoing weight and that output's delta.

Output 1

Output delta−0.55

×

Old outgoing weight0.5

=

Path contribution−0.275

↓

Add every path contribution−0.275Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

Contribution from output 1.

LaTeX: C_1=(0.5)\times(-0.55)\approx -0.275

Accumulate the contributions, then apply the hidden activation derivative below.

LaTeX: \text{Sum}=C_1\approx -0.275

LaTeX: \delta_{h_2} = -0.275 \times 1 = -0.275

Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.

x1 → h1:

Sending input1x1

×

Receiving delta−0.22δ at h1

=

Connection gradient−0.22x1 → h1

Now update this weight

Old weight0.2

→

New weight0.222

Subtract η × gradient = −0.022. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

LaTeX: g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1

LaTeX: g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222

x2 → h1:

Sending input2x2

×

Receiving delta−0.22δ at h1

=

Connection gradient−0.44x2 → h1

Now update this weight

Old weight0.1

→

New weight0.144

Subtract η × gradient = −0.044. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

LaTeX: g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2

LaTeX: g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144

x1 → h2:

Sending input1x1

×

Receiving delta−0.275δ at h2

=

Connection gradient−0.275x1 → h2

Now update this weight

Old weight0.3

→

New weight0.3275

Subtract η × gradient = −0.0275. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

LaTeX: g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1

LaTeX: g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275

x2 → h2:

Sending input2x2

×

Receiving delta−0.275δ at h2

=

Connection gradient−0.55x2 → h2

Now update this weight

Old weight−0.1

→

New weight−0.045

Subtract η × gradient = −0.055. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

LaTeX: g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2

LaTeX: g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045

Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.

LaTeX: b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000

LaTeX: b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000

LaTeX: b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000

These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.

Drill 12 Two output sigmoid backpropagation

Question

Question. Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters.

Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4

Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass.

Answer — step by step

Given

Input x = [0.2, 0.4].

W₁ is printed with the question.

b₁ = [0.1, 0.1].

W₂ is printed with the question.

b₂ = [0, 0.1].

Hidden and output activations are sigmoid.

Target y = [1, 0].

Learning rate η = 0.2.

Use half summed squared loss and update all parameters.

To find

Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.

Formula — which one and why

Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.

x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.

LaTeX: z_h=W_1x+b_1

LaTeX: h=f_1(z_h)

LaTeX: z_o=W_2h+b_2

LaTeX: o=f_2(z_o)

Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.

Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.

LaTeX: L=\frac12\sum_j(o_j-y_j)^2

LaTeX: \frac{\partial L}{\partial o_j}=o_j-y_j

Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

LaTeX: p=\frac{1}{1+e^{-z}}

LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Formulas used before substituting numbers

For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.

LaTeX: z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2

Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.

LaTeX: \delta_{o_k}=(o_k-y_k)f'(z_{o_k})

LaTeX: \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j})

The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.

LaTeX: \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j

LaTeX: w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j

For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.

Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.

Hidden neuron 1:

LaTeX: z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2

LaTeX: h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997

Hidden neuron 2:

LaTeX: z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18

LaTeX: h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892

Output neuron 1:

LaTeX: z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891

LaTeX: o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171

Output neuron 2:

LaTeX: z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756

LaTeX: o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848

Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.

Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide.

Output delta 1:

LaTeX: \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988

The derivative is o(1−o).

Output delta 2:

LaTeX: \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863

The derivative is o(1−o).

Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.

h1 → o1:

Sending activation0.549833997h1

×

Receiving delta−0.121439988δ at o1

=

Connection gradient−0.066771834h1 → o1

Now update this weight

Old weight0.2

→

New weight0.213354367

Subtract η × gradient = −0.013354367. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

LaTeX: g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1

LaTeX: g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367

h2 → o1:

Sending activation0.544878892h2

×

Receiving delta−0.121439988δ at o1

=

Connection gradient−0.066170086h2 → o1

Now update this weight

Old weight−0.1

→

New weight−0.086765983

Subtract η × gradient = −0.013234017. This weight increases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

LaTeX: g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2

LaTeX: g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983

h1 → o2:

Sending activation0.549833997h1

×

Receiving delta0.145933863δ at o2

=

Connection gradient0.080239399h1 → o2

Now update this weight

Old weight0.3

→

New weight0.28395212

Subtract η × gradient = 0.01604788. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

LaTeX: g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1

LaTeX: g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212

h2 → o2:

Sending activation0.544878892h2

×

Receiving delta0.145933863δ at o2

=

Connection gradient0.079516282h2 → o2

Now update this weight

Old weight0.4

→

New weight0.384096744

Subtract η × gradient = 0.015903256. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies its activation.

LaTeX: g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2

LaTeX: g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744

Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.

Hidden 1 downstream contribution:

Calculate each outgoing path using the old outgoing weight and that output's delta.

Output 1

Output delta−0.121439988

×

Old outgoing weight0.2

=

Path contribution−0.024287998

Output 2

Output delta0.145933863

×

Old outgoing weight0.3

=

Path contribution0.043780159

↓

Add every path contribution0.019492161Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

Contribution from output 1.

LaTeX: C_1=(0.2)\times(-0.121439988)\approx -0.024287998

Contribution from output 2.

LaTeX: C_2=(0.3)\times(0.145933863)\approx 0.043780159

Accumulate the contributions, then apply the hidden activation derivative below.

LaTeX: \text{Sum}=C_1+C_2\approx 0.019492161

LaTeX: \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633

Hidden 2 downstream contribution:

Calculate each outgoing path using the old outgoing weight and that output's delta.

Output 1

Output delta−0.121439988

×

Old outgoing weight−0.1

=

Path contribution0.012143999

Output 2

Output delta0.145933863

×

Old outgoing weight0.4

=

Path contribution0.058373545

↓

Add every path contribution0.070517544Then multiply by the hidden activation derivative.

Multiply within each path. Add the resulting contributions at the shared hidden neuron.

Contribution from output 1.

LaTeX: C_1=(-0.1)\times(-0.121439988)\approx 0.012143999

Contribution from output 2.

LaTeX: C_2=(0.4)\times(0.145933863)\approx 0.058373545

Accumulate the contributions, then apply the hidden activation derivative below.

LaTeX: \text{Sum}=C_1+C_2\approx 0.070517544

LaTeX: \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356

Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.

x1 → h1:

Sending input0.2x1

×

Receiving delta0.004824633δ at h1

=

Connection gradient0.000964927x1 → h1

Now update this weight

Old weight0.1

→

New weight0.099807015

Subtract η × gradient = 0.000192985. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

LaTeX: g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1

LaTeX: g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015

x2 → h1:

Sending input0.4x2

×

Receiving delta0.004824633δ at h1

=

Connection gradient0.001929853x2 → h1

Now update this weight

Old weight0.2

→

New weight0.199614029

Subtract η × gradient = 0.000385971. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

LaTeX: g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2

LaTeX: g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029

x1 → h2:

Sending input0.2x1

×

Receiving delta0.017487356δ at h2

=

Connection gradient0.003497471x1 → h2

Now update this weight

Old weight−0.2

→

New weight−0.200699494

Subtract η × gradient = 0.000699494. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

LaTeX: g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1

LaTeX: g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494

x2 → h2:

Sending input0.4x2

×

Receiving delta0.017487356δ at h2

=

Connection gradient0.006994942x2 → h2

Now update this weight

Old weight0.3

→

New weight0.298601012

Subtract η × gradient = 0.001398988. This weight decreases.

The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.

The receiving neuron supplies the delta; the sending neuron supplies the input value.

LaTeX: g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2

LaTeX: g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942

Apply the learning rate to this gradient and subtract the result from the old weight.

LaTeX: W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012

Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.

LaTeX: b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073

LaTeX: b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529

LaTeX: b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998

LaTeX: b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227

These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.

Sigmoid and its derivative

Sigmoid takes a raw score and changes it to a number between 0 and 1. Its derivative is a different number: it tells backpropagation how sensitive that output is to its raw score. The complete example below calculates both and shows where each is used.

LaTeX: \sigma(z)=\frac{1}{1+e^{-z}}

LaTeX: \sigma\prime(z)=\sigma(z)[1-\sigma(z)]

Complete sigmoid training example

Question. A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update.

Given

x = 2; w = 0; b = 0; y = 1.

Sigmoid output; half squared loss; η = 0.1.

To find

Find the weight-gradient sign, exact weight update and resulting weight.

Formula — which one and why

Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.

LaTeX: z=\sum_iw_ix_i+b

LaTeX: a=f(z)

Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

LaTeX: p=\frac{1}{1+e^{-z}}

LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.

LaTeX: L=\frac12\sum_j(o_j-y_j)^2

LaTeX: \frac{\partial L}{\partial o_j}=o_j-y_j

Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Step 1 forward: net = 0(2) + 0 = 0; sigmoid output = 0.5.

Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input.

Step 2 backward: prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125.

Step 3 gradients and update: weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree.

Drill 10 Sigmoid neuron gradient

Answer — step by step

Given

x = 2; w = 0.5; b = 0; target y = 1.

Activation: sigmoid; loss: half squared error.

Learning rate η = 0.1.

To find

Find the output, delta, weight/bias gradients and one update.

Formula — which one and why

Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.

LaTeX: z=\sum_iw_ix_i+b

LaTeX: a=f(z)

Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

LaTeX: p=\frac{1}{1+e^{-z}}

LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.

LaTeX: L=\frac12\sum_j(o_j-y_j)^2

LaTeX: \frac{\partial L}{\partial o_j}=o_j-y_j

Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Meaning and formulas

Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly.

LaTeX: z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2

The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z.

LaTeX: \delta=(p-y)p(1-p)

LaTeX: \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta

LaTeX: w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta

Step 1: weighted sum and sigmoid

LaTeX: z=0.5\times2+0=1

LaTeX: p=\frac1{1+e^{-1}}=0.731058579

Step 2: error and activation slope

LaTeX: p-y=0.731058579-1=-0.268941421

LaTeX: p(1-p)=0.731058579\times0.268941421=0.196611933

Step 3: multiply the linked effects to obtain delta

LaTeX: \delta=(-0.268941421)\times0.196611933=-0.052877093

Step 4: multiply delta by the input to each parameter

The weight receives x = 2. The bias effectively receives the constant input 1.

LaTeX: \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186

LaTeX: \frac{\partial L}{\partial b}=-0.052877093

Step 5: subtract learning rate times each gradient

LaTeX: w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419

LaTeX: b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709

Final answer: new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here.

sigmoid example 1

Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.

Given

z = -2; activation = sigmoid.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\frac1{1+e^{-z}}

LaTeX: f\prime(z)=a(1-a)

Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).

Solution — step by step

Step 1: negate the whole input.

LaTeX: -z=-(-2)=2

Step 2: calculate the exponential.

LaTeX: e^{2}\approx7.389056099

Step 3: add one in the denominator.

LaTeX: 1+7.389056099\approx8.389056099

Step 4: divide one by the complete denominator.

LaTeX: a=\frac1{8.389056099}\approx0.119202922

Step 5: use the activated output in the slope.

LaTeX: f\prime(-2)=(0.119202922)(1-0.119202922)\approx0.104993585

Final answer: a ≈ 0.119202922. Derivative ≈ 0.104993585.

sigmoid example 2

Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.

Given

z = 0; activation = sigmoid.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\frac1{1+e^{-z}}

LaTeX: f\prime(z)=a(1-a)

Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).

Solution — step by step

Step 1: negate the whole input.

LaTeX: -z=-(0)=0

Step 2: calculate the exponential.

LaTeX: e^{0}\approx1.000000000

Step 3: add one in the denominator.

LaTeX: 1+1.000000000\approx2.000000000

Step 4: divide one by the complete denominator.

LaTeX: a=\frac1{2.000000000}\approx0.500000000

Step 5: use the activated output in the slope.

LaTeX: f\prime(0)=(0.500000000)(1-0.500000000)\approx0.250000000

Final answer: a ≈ 0.500000000. Derivative ≈ 0.250000000.

sigmoid example 3

Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.

Given

z = 2; activation = sigmoid.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\frac1{1+e^{-z}}

LaTeX: f\prime(z)=a(1-a)

Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).

Solution — step by step

Step 1: negate the whole input.

LaTeX: -z=-(2)=-2

Step 2: calculate the exponential.

LaTeX: e^{-2}\approx0.135335283

Step 3: add one in the denominator.

LaTeX: 1+0.135335283\approx1.135335283

Step 4: divide one by the complete denominator.

LaTeX: a=\frac1{1.135335283}\approx0.880797078

Step 5: use the activated output in the slope.

LaTeX: f\prime(2)=(0.880797078)(1-0.880797078)\approx0.104993585

Final answer: a ≈ 0.880797078. Derivative ≈ 0.104993585.

Tanh and its derivative

Tanh can produce a negative or positive output. In the formula, calculate the exponential first, then form the top and bottom of the fraction separately. The derivative uses the activated answer, not the raw input.

LaTeX: \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1}

LaTeX: \tanh\prime(z)=1-\tanh^2(z)

Question. Calculate tanh(0.5) and its derivative with respect to the input.

Given

z = 0.5.

To find

Find tanh(z) and its derivative.

Formula — which one and why

Tanh and its slope. Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a².

z = value entering tanh; a = its output. A prime means derivative, or local rate of change.

LaTeX: a=\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}}

LaTeX: \frac{da}{dz}=1-a^2

Yaad rakhne ka cue: Tanh signed output deta hai; slope = 1 minus output ka square.

Solution — step by step

Step 1: identify the input. The input z is 0.5. In this version of the tanh formula, the exponential uses 2z.

LaTeX: 2z=2(0.5)=1,\qquad e^{2z}=e^1\approx2.718281828

Step 2: calculate the numerator. Subtract 1 from that exponential.

LaTeX: e^{2z}-1=2.718281828-1=1.718281828

Step 3: calculate the denominator. Add 1 to the same exponential.

LaTeX: e^{2z}+1=2.718281828+1=3.718281828

Step 4: divide the top by the bottom.

LaTeX: a=\tanh(0.5)=\frac{1.718281828}{3.718281828}\approx0.462117157

Step 5: calculate the local derivative. Square the output a and subtract it from 1. Keep the calculator's unrounded result.

LaTeX: a^2\approx0.213552267,\qquad1-a^2\approx0.786447733

Final answer: tanh output ≈ 0.462117157; local derivative ≈ 0.786447733. The output becomes the next layer's input; the derivative is used when carrying a gradient backward.

tanh example 1

Question. For preactivation z = -2, calculate the tanh output and state its derivative.

Given

z = -2; activation = tanh.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\frac{e^{2z}-1}{e^{2z}+1}

LaTeX: f\prime(z)=1-a^2

Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.

Solution — step by step

Step 1: double z before exponentiation.

LaTeX: 2z=2(-2)=-4

Step 2: calculate the exponential.

LaTeX: e^{-4}\approx0.018315639

Step 3: calculate numerator and denominator separately.

LaTeX: 0.018315639-1\approx-0.981684361

LaTeX: 0.018315639+1\approx1.018315639

Step 4: divide.

LaTeX: a=\frac{-0.981684361}{1.018315639}\approx-0.964027580

Step 5: square the activated output for its slope.

LaTeX: f\prime(-2)=1-(-0.964027580)^2\approx0.070650825

Final answer: a ≈ -0.964027580. Derivative ≈ 0.070650825.

tanh example 2

Question. For preactivation z = 0, calculate the tanh output and state its derivative.

Given

z = 0; activation = tanh.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\frac{e^{2z}-1}{e^{2z}+1}

LaTeX: f\prime(z)=1-a^2

Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.

Solution — step by step

Step 1: double z before exponentiation.

LaTeX: 2z=2(0)=0

Step 2: calculate the exponential.

LaTeX: e^{0}\approx1.000000000

Step 3: calculate numerator and denominator separately.

LaTeX: 1.000000000-1\approx0.000000000

LaTeX: 1.000000000+1\approx2.000000000

Step 4: divide.

LaTeX: a=\frac{0.000000000}{2.000000000}\approx0.000000000

Step 5: square the activated output for its slope.

LaTeX: f\prime(0)=1-(0.000000000)^2\approx1.000000000

Final answer: a ≈ 0.000000000. Derivative ≈ 1.000000000.

tanh example 3

Question. For preactivation z = 2, calculate the tanh output and state its derivative.

Given

z = 2; activation = tanh.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\frac{e^{2z}-1}{e^{2z}+1}

LaTeX: f\prime(z)=1-a^2

Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.

Solution — step by step

Step 1: double z before exponentiation.

LaTeX: 2z=2(2)=4

Step 2: calculate the exponential.

LaTeX: e^{4}\approx54.598150033

Step 3: calculate numerator and denominator separately.

LaTeX: 54.598150033-1\approx53.598150033

LaTeX: 54.598150033+1\approx55.598150033

Step 4: divide.

LaTeX: a=\frac{53.598150033}{55.598150033}\approx0.964027580

Step 5: square the activated output for its slope.

LaTeX: f\prime(2)=1-(0.964027580)^2\approx0.070650825

Final answer: a ≈ 0.964027580. Derivative ≈ 0.070650825.

ReLU and its derivative

ReLU compares its input with zero and keeps the larger value. Thus a negative input becomes zero and a positive input passes unchanged. Its slope is zero on the negative side and one on the positive side.

LaTeX: f(z)=\max(0,z)

LaTeX: f\prime(z)=\begin{cases}1&z>0\\0&z<0\end{cases}

Drill 8 Activation contrast

Answer — step by step

Given

Inputs z = [−3,0,2].

Leaky-ReLU negative slope α = 0.1.

For the sigmoid derivative, use z = 0.

To find

Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero.

Formula — which one and why

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

LaTeX: \operatorname{ReLU}(z)=\max(0,z)

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

LaTeX: p=\frac{1}{1+e^{-z}}

LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Solution — step by step

Formulas and their meanings

ReLU compares the input with zero and returns the larger value.

LaTeX: \operatorname{ReLU}(z)=\max(0,z)

Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative.

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Step 1: apply ReLU separately to all three inputs

LaTeX: \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2

ReLU output = [0, 0, 2].

Step 2: use α = 0.1 only on the negative input

LaTeX: 0.1\times(-3)=-0.3

The zero and positive entries remain 0 and 2. Leaky ReLU output = [−0.3, 0, 2].

Step 3: calculate sigmoid before its derivative

LaTeX: \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)]

LaTeX: \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5

LaTeX: \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25

The sigmoid value is 0.5 but its slope is 0.25. They are different quantities.

ReLU example 1

Question. For preactivation z = -2, calculate the ReLU output and state its derivative.

Given

z = -2; activation = ReLU.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

Solution — step by step

Step 1: select the branch from the sign of z.

-2 is negative, so use the negative branch.

Step 2: calculate the output.

LaTeX: a=0

Step 3: state the slope on that branch.

LaTeX: f\prime(-2)=0

Final answer: a ≈ 0.000000000. Derivative ≈ 0.000000000.

ReLU example 2

Question. For preactivation z = 0, calculate the ReLU output and state its derivative.

Given

z = 0; activation = ReLU.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

Solution — step by step

Step 1: select the branch from the sign of z.

Both branches meet at zero. The value is zero, but the left and right slopes differ.

Step 2: calculate the output.

LaTeX: a=0

Step 3: state the slope on that branch.

The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.

Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.

ReLU example 3

Question. For preactivation z = 2, calculate the ReLU output and state its derivative.

Given

z = 2; activation = ReLU.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

Solution — step by step

Step 1: select the branch from the sign of z.

2 is positive, so retain it.

Step 2: calculate the output.

LaTeX: a=2

Step 3: state the slope on that branch.

LaTeX: f\prime(2)=1

Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.

Leaky ReLU

Leaky ReLU changes only the negative side: multiply a negative input by the small slope α. That negative sign remains. The value α is supplied by the question or chosen as a model setting; it is not the sigmoid function.

LaTeX: f(z)=\begin{cases}z&z\geq0\\az&z<0\end{cases}

Drill 8 Activation contrast

Answer — step by step

Given

Inputs z = [−3,0,2].

Leaky-ReLU negative slope α = 0.1.

For the sigmoid derivative, use z = 0.

To find

Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero.

Formula — which one and why

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

LaTeX: \operatorname{ReLU}(z)=\max(0,z)

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

LaTeX: p=\frac{1}{1+e^{-z}}

LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Solution — step by step

Formulas and their meanings

ReLU compares the input with zero and returns the larger value.

LaTeX: \operatorname{ReLU}(z)=\max(0,z)

Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative.

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Step 1: apply ReLU separately to all three inputs

LaTeX: \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2

ReLU output = [0, 0, 2].

Step 2: use α = 0.1 only on the negative input

LaTeX: 0.1\times(-3)=-0.3

The zero and positive entries remain 0 and 2. Leaky ReLU output = [−0.3, 0, 2].

Step 3: calculate sigmoid before its derivative

LaTeX: \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)]

LaTeX: \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5

LaTeX: \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25

The sigmoid value is 0.5 but its slope is 0.25. They are different quantities.

Leaky ReLU example 1

Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.

Given

z = -2; activation = Leaky ReLU.

α = 0.1.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases}

Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

Solution — step by step

Step 1: select the branch from the sign of z.

-2 is negative, so use the negative branch.

Step 2: calculate the output.

LaTeX: a=-0.2

Step 3: state the slope on that branch.

LaTeX: f\prime(-2)=0.1

Final answer: a ≈ -0.200000000. Derivative ≈ 0.100000000.

Leaky ReLU example 2

Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.

Given

z = 0; activation = Leaky ReLU.

α = 0.1.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases}

Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

Solution — step by step

Step 1: select the branch from the sign of z.

Both branches meet at zero. The value is zero, but the left and right slopes differ.

Step 2: calculate the output.

LaTeX: a=0

Step 3: state the slope on that branch.

The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.

Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.

Leaky ReLU example 3

Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.

Given

z = 2; activation = Leaky ReLU.

α = 0.1.

To find

The output a and the derivative with respect to z.

Formula — which one and why

Use the activation named in the question. The derivative is a second calculation required for a backward pass.

z is the raw sum; a is the activated output; f′(z) is the local slope.

LaTeX: a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases}

Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.

Solution — step by step

Step 1: select the branch from the sign of z.

2 is positive, so retain it.

Step 2: calculate the output.

LaTeX: a=2

Step 3: state the slope on that branch.

LaTeX: f\prime(2)=1

Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.

Stable softmax

Softmax turns several competing class scores into probabilities that add to one. Exponentiate each score, add those exponential values, and divide each one by their common total. Subtracting the same maximum score first gives the same probabilities with safer arithmetic.

LaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}

LaTeX: c=\max_j z_j

Drill 9 Softmax and true class loss

Answer — step by step

Given

Logits = [1,2,0].

One-hot target = [0,1,0]: class 2 is correct.

To find

Find all three softmax probabilities and CCE.

Formula — which one and why

Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.

zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.

LaTeX: M=\max_j z_j

LaTeX: p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}

Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.

Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.

yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.

LaTeX: L=-\sum_j y_j\ln p_j

LaTeX: L=-\ln p_c

Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.

Solution — step by step

Meaning and formulas

The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities.

LaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j

Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials.

LaTeX: L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i

Step 1: subtract the largest score

The largest of [1,2,0] is 2.

LaTeX: [1-2,\;2-2,\;0-2]=[-1,0,-2]

Step 2: calculate the three exponentials

LaTeX: e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283

Step 3: add them to get one common denominator

LaTeX: D=0.367879441+1+0.135335283=1.503214724

Step 4: divide each exponential by this denominator

LaTeX: p_1=\frac{0.367879441}{1.503214724}=0.244728471

LaTeX: p_2=\frac{1}{1.503214724}=0.665240956

LaTeX: p_3=\frac{0.135335283}{1.503214724}=0.090030573

The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability.

Step 5: use the probability of the correct class in CCE

LaTeX: L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2

LaTeX: L=-\ln(0.665240956)=0.407605964

Final answer: probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606.

Softmax for [0, 0, 0]

Question. Find stable softmax probabilities for logits [0, 0, 0].

Given

Logits = [0, 0, 0]; there are three mutually exclusive classes.

To find

All three class probabilities.

Formula — which one and why

Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.

A logit is a raw class score. c is the largest score; pᵢ is probability for class i.

LaTeX: c=\max_j z_j

LaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}

Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.

Solution — step by step

Step 1: subtract the largest score from every entry.

LaTeX: z-c=\begin{bmatrix}0&0&0\end{bmatrix}

Step 2: exponentiate shifted entry 1.

LaTeX: e^{0}\approx1.000000000

Step 3: exponentiate shifted entry 2.

LaTeX: e^{0}\approx1.000000000

Step 4: exponentiate shifted entry 3.

LaTeX: e^{0}\approx1.000000000

Step 5: add all exponentials into one denominator.

LaTeX: D=1.000000000+1.000000000+1.000000000\approx3.000000000

Step 6: divide entry 1 by the same denominator.

LaTeX: p_1=\frac{1.000000000}{3.000000000}\approx0.333333333

Step 7: divide entry 2 by the same denominator.

LaTeX: p_2=\frac{1.000000000}{3.000000000}\approx0.333333333

Step 8: divide entry 3 by the same denominator.

LaTeX: p_3=\frac{1.000000000}{3.000000000}\approx0.333333333

Final answer: Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding.

Softmax for [1, 2, 3]

Question. Find stable softmax probabilities for logits [1, 2, 3].

Given

Logits = [1, 2, 3]; there are three mutually exclusive classes.

To find

All three class probabilities.

Formula — which one and why

Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.

A logit is a raw class score. c is the largest score; pᵢ is probability for class i.

LaTeX: c=\max_j z_j

LaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}}

Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.

Solution — step by step

Step 1: subtract the largest score from every entry.

LaTeX: z-c=\begin{bmatrix}-2&-1&0\end{bmatrix}

Step 2: exponentiate shifted entry 1.

LaTeX: e^{-2}\approx0.135335283

Step 3: exponentiate shifted entry 2.

LaTeX: e^{-1}\approx0.367879441

Step 4: exponentiate shifted entry 3.

LaTeX: e^{0}\approx1.000000000

Step 5: add all exponentials into one denominator.

LaTeX: D=0.135335283+0.367879441+1.000000000\approx1.503214724

Step 6: divide entry 1 by the same denominator.

LaTeX: p_1=\frac{0.135335283}{1.503214724}\approx0.090030573

Step 7: divide entry 2 by the same denominator.

LaTeX: p_2=\frac{0.367879441}{1.503214724}\approx0.244728471

Step 8: divide entry 3 by the same denominator.

LaTeX: p_3=\frac{1.000000000}{1.503214724}\approx0.665240956

Final answer: Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding.

Squared and absolute losses

SSE, half SSE and MSE start with the same errors but apply different final divisions. First understand what an error and its square mean. The full question below keeps all three answers separate.

LaTeX: \mathrm{SSE}=\sum_i(o_i-y_i)^2

LaTeX: L_{\mathrm{half}}=\frac{\mathrm{SSE}}2

LaTeX: L_{\mathrm{MSE}}=\frac{\mathrm{SSE}}n

LaTeX: L_{\mathrm{MAE}}=\frac1n\sum_i|o_i-y_i|

Loss: how one wrong prediction becomes a number

Definition. A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative.

LaTeX: e_i=\hat y_i-y_i,\qquad \mathrm{SSE}=\sum_{i=1}^{n}e_i^2,\qquad L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2,\qquad \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2

For this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective.

Derivatives and their scaling

When differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual:

LaTeX: \frac{\partial\mathrm{SSE}}{\partial\hat y_j}=2e_j,\qquad \frac{\partial L_{\mathrm{half}}}{\partial\hat y_j}=e_j,\qquad \frac{\partial\mathrm{MSE}}{\partial\hat y_j}=\frac{2}{n}e_j

The factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight.

Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.

Given

Predictions = [0.2,0.7,0.4].

Targets = [0,1,1]; n = 3 scalar predictions.

To find

Find SSE, half SSE and MSE, then all three prediction-gradient lists.

Formula — which one and why

Squared-error objectives. Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.

ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.

LaTeX: e_i=\hat y_i-y_i

LaTeX: \mathrm{SSE}=\sum_i e_i^2

LaTeX: L_{\mathrm{half}}=\frac12\sum_i e_i^2

LaTeX: \mathrm{MSE}=\frac1n\sum_i e_i^2

Yaad rakhne ka cue: Pehle error square; phir question ke hisaab se sum, half ya mean.

Solution — step by step

Step 1: calculate the three signed prediction errors.

LaTeX: e=[0.2-0,\ 0.7-1,\ 0.4-1]=[0.2,-0.3,-0.6]

Step 2: square every error.

LaTeX: e\odot e=[0.04,0.09,0.36]

Step 3: add the three squared errors.

LaTeX: \mathrm{SSE}=0.04+0.09+0.36=0.49

Step 4: apply the requested loss divisors.

LaTeX: L_{\mathrm{half}}=\frac{0.49}{2}=0.245,\qquad \mathrm{MSE}=\frac{0.49}{3}\approx0.163333333

Step 5: multiply every error by 2 for the SSE gradient.

LaTeX: \nabla_{\hat y}\mathrm{SSE}=[0.4,-0.6,-1.2]

Step 6: use each error itself for the half-SSE gradient.

LaTeX: \nabla_{\hat y}L_{\mathrm{half}}=[0.2,-0.3,-0.6]

Step 7: multiply every error by 2/3 for the MSE gradient.

LaTeX: \nabla_{\hat y}\mathrm{MSE}=\frac23[0.2,-0.3,-0.6]=[0.133333333,-0.2,-0.4]

The residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate.

Drill 2 Half sum versus mean

Answer — step by step

Given

Predictions = [0.2, 0.7, 0.4].

Targets = [0, 1, 1]; number of outputs n = 3.

To find

Find SSE, half SSE, MSE and the prediction gradients requested in the question.

Formula — which one and why

Squared-error objectives. Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.

ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.

LaTeX: e_i=\hat y_i-y_i

LaTeX: \mathrm{SSE}=\sum_i e_i^2

LaTeX: L_{\mathrm{half}}=\frac12\sum_i e_i^2

LaTeX: \mathrm{MSE}=\frac1n\sum_i e_i^2

Yaad rakhne ka cue: Pehle error square; phir question ke hisaab se sum, half ya mean.

Solution — step by step

Given values and loss definitions

There are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i.

LaTeX: \mathrm{SSE}=\sum_{i=1}^{n}e_i^2

LaTeX: L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}2

LaTeX: \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}n

SSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively.

Step 1: calculate each residual and its square

Pair	Prediction ŷᵢ	Target yᵢ	Residual ŷᵢ − yᵢ	Squared residual
1	0.2	0	0.2 − 0 = 0.2	(0.2)² = 0.04
2	0.7	1	0.7 − 1 = −0.3	(−0.3)² = 0.09
3	0.4	1	0.4 − 1 = −0.6	(−0.6)² = 0.36

Step 2: sum the squared residuals

LaTeX: \mathrm{SSE}=0.04+0.09+0.36=0.49

Square each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel.

Step 3: apply each requested normalization

For half SSE, the denominator is 2:

LaTeX: L_{\mathrm{half}}=\frac{0.49}{2}=0.245

For MSE, the denominator is the number of pairs, n = 3:

LaTeX: \mathrm{MSE}=\frac{0.49}{3}=0.163333333\ldots

Final answer: SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question.

Absolute error uses a different operation

Question. Targets are [2, 4] and predictions are [1, 6]. Find mean absolute error.

Given

Targets = [2,4]; predictions = [1,6]; n = 2.

To find

Find mean absolute error.

Formula — which one and why

Mean absolute error. Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes.

n = number of predictions. |e| means the nonnegative magnitude of error e.

LaTeX: \mathrm{MAE}=\frac1n\sum_i|\hat y_i-y_i|

Yaad rakhne ka cue: MAE mein magnitude; MSE mein square.

Solution — step by step

The signed errors are 1 − 2 = −1 and 6 − 4 = 2. Absolute value keeps the size of a number without its sign, so their absolute errors are 1 and 2. Add them to obtain 3. There are two prediction–target pairs, so divide by 2.

LaTeX: \mathrm{MAE}=\frac{|1-2|+|6-4|}{2}=\frac{1+2}{2}=1.5

Final answer: MAE = 1.5. Do not square the errors for MAE.

Binary cross entropy

Binary cross-entropy measures how much probability was assigned to the correct binary answer. A target of 1 keeps the −ln(p) term; a target of 0 keeps the −ln(1−p) term. The factors y and 1−y select the appropriate term.

LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

Drill 7 Sigmoid with binary cross entropy

Answer — step by step

Given

Logit z = 1; binary target y = 1.

Output: sigmoid; loss: binary cross-entropy.

To find

Find probability, BCE and the loss derivative with respect to z.

Formula — which one and why

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

LaTeX: p=\frac{1}{1+e^{-z}}

LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Binary cross-entropy. For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.

y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.

LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

LaTeX: \frac{\partial L}{\partial z}=p-y

Yaad rakhne ka cue: Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).

Solution — step by step

Meaning and formulas

A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome.

LaTeX: p=\frac1{1+e^{-z}}

LaTeX: L=-\left[y\ln p+(1-y)\ln(1-p)\right]

For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form:

LaTeX: \frac{\partial L}{\partial z}=p-y

Step 1: convert logit 1 into probability

LaTeX: e^{-1}\approx0.367879441

LaTeX: 1+e^{-1}\approx1.367879441

LaTeX: p=\frac1{1.367879441}\approx0.731058579

Step 2: substitute the target in the full loss formula

LaTeX: L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right]

The second term has multiplier zero, so it contributes zero.

LaTeX: L=-\ln(0.731058579)\approx0.313261688

Step 3: calculate the combined gradient

LaTeX: \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421

Final answer: probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice.

Binary cross-entropy: target 1, predicted positive-class probability 0.8

Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.

Given

Target y = 1; p = P(y=1) = 0.8; use natural log ln.

To find

The binary cross-entropy loss for this one example.

Formula — which one and why

The target selects the probability assigned to the correct class. Take its negative natural logarithm.

p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.

LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.

Solution — step by step

Step 1: identify the correct-class probability.

LaTeX: p=0.8

Step 2: evaluate its natural logarithm.

LaTeX: \ln(0.8)\approx-0.223143551

Step 3: apply the minus sign outside the logarithm.

LaTeX: L=-(-0.223143551)=0.223143551

Final answer: Loss ≈ 0.223143551 nats.

Binary cross-entropy: target 0, predicted positive-class probability 0.8

Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.

Given

Target y = 0; p = P(y=1) = 0.8; use natural log ln.

To find

The binary cross-entropy loss for this one example.

Formula — which one and why

The target selects the probability assigned to the correct class. Take its negative natural logarithm.

p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.

LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.

Solution — step by step

Step 1: identify the correct-class probability.

LaTeX: 1-p=1-0.8=0.2

Step 2: evaluate its natural logarithm.

LaTeX: \ln(0.2)\approx-1.609437912

Step 3: apply the minus sign outside the logarithm.

LaTeX: L=-(-1.609437912)=1.609437912

Final answer: Loss ≈ 1.609437912 nats.

Binary cross-entropy: target 1, predicted positive-class probability 0.5

Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.

Given

Target y = 1; p = P(y=1) = 0.5; use natural log ln.

To find

The binary cross-entropy loss for this one example.

Formula — which one and why

The target selects the probability assigned to the correct class. Take its negative natural logarithm.

p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.

LaTeX: L=-[y\ln p+(1-y)\ln(1-p)]

Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.

Solution — step by step

Step 1: identify the correct-class probability.

LaTeX: p=0.5

Step 2: evaluate its natural logarithm.

LaTeX: \ln(0.5)\approx-0.693147181

Step 3: apply the minus sign outside the logarithm.

LaTeX: L=-(-0.693147181)=0.693147181

Final answer: Loss ≈ 0.693147181 nats.

Categorical cross entropy

For one-hot multiclass targets, only one target entry is 1. The other entries are zero, so the loss becomes minus the logarithm of the true class probability. The worked question first calculates every softmax probability, then selects the correct one.

LaTeX: H(p,q)=-\sum_i p_i\ln q_i

LaTeX: \log_2q=\frac{\ln q}{\ln2}

LaTeX: L=-\sum_i y_i\ln p_i=-\ln p_{true}

Cross-entropy for probability distributions

Definition. For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities.

LaTeX: H(p,q)=-\sum_{i=1}^{K}p_i\log q_i

Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms.

Logarithm base and units

Natural logarithms produce nats; base-2 logarithms produce bits. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2:

LaTeX: \log_2 q=\frac{\ln q}{\ln 2}

For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities.

A target distribution with two nonzero entries

Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.

Given

A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5].

Calculate cross-entropy using natural logarithms and using base-2 logarithms.

To find

Find cross-entropy in the requested units.

Formula — which one and why

Cross-entropy between distributions. Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute.

pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits.

LaTeX: H(p,q)=-\sum_i p_i\ln q_i

LaTeX: H_2(p,q)=\frac{H(p,q)}{\ln2}

Yaad rakhne ka cue: Target probability batati hai kis log-loss ka kitna hissa lena hai.

Solution — step by step

Step 1: write the two terms. Both target entries are nonzero, so neither term can be discarded.

LaTeX: H(p,q)=-[0.75\ln(0.5)+0.25\ln(0.5)]

Step 2: calculate the logarithms.

LaTeX: \ln(0.5)\approx-0.693147181

Step 3: weight each negative logarithm by its target probability.

LaTeX: -0.75\ln(0.5)\approx0.519860385

LaTeX: -0.25\ln(0.5)\approx0.173286795

Step 4: add the contributions, retaining calculator precision.

LaTeX: H(p,q)\approx0.519860385+0.173286795\approx0.693147181\ \text{nats}

Step 5: repeat with base 2. Since 2⁻¹ = 0.5, log₂(0.5) = −1.

LaTeX: H_2(p,q)=-[0.75(-1)+0.25(-1)]=1\ \text{bit}

Final answer: 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale.

Cross-entropy, entropy and relative entropy

These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio.

LaTeX: H(p)=-\sum_i p_i\ln p_i

LaTeX: D_{\mathrm{KL}}(p\Vert q)=\sum_i p_i\ln\frac{p_i}{q_i}=H(p,q)-H(p)

For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus.

Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.

Given

For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.

To find

Find entropy and KL divergence in nats.

Formula — which one and why

Entropy and relative entropy. Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p.

p = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence.

LaTeX: H(p)=-\sum_i p_i\ln p_i

LaTeX: D_{\mathrm{KL}}(p\Vert q)=H(p,q)-H(p)

Yaad rakhne ka cue: KL = cross-entropy minus target ki own entropy.

Solution — step by step

Step 1: calculate the target entropy.

LaTeX: H(p)=-[0.75\ln(0.75)+0.25\ln(0.25)]

LaTeX: \ln(0.75)\approx-0.287682072,\qquad\ln(0.25)\approx-1.386294361

LaTeX: H(p)\approx0.215761554+0.346573590=0.562335145

Step 2: subtract target entropy from cross-entropy.

LaTeX: D_{\mathrm{KL}}(p\Vert q)\approx0.693147181-0.562335145=0.130812036

Final answer: H(p) ≈ 0.562335 nats and DKL(p ∥ q) ≈ 0.130812 nats. For a one-hot target H(p) = 0, so cross-entropy and this KL divergence coincide; for this soft target they differ.

Drill 9 Softmax and true class loss

Answer — step by step

Given

Logits = [1,2,0].

One-hot target = [0,1,0]: class 2 is correct.

To find

Find all three softmax probabilities and CCE.

Formula — which one and why

Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.

zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.

LaTeX: M=\max_j z_j

LaTeX: p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}}

Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.

Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.

yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.

LaTeX: L=-\sum_j y_j\ln p_j

LaTeX: L=-\ln p_c

Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.

Solution — step by step

Meaning and formulas

The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities.

LaTeX: p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j

Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials.

LaTeX: L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i

Step 1: subtract the largest score

The largest of [1,2,0] is 2.

LaTeX: [1-2,\;2-2,\;0-2]=[-1,0,-2]

Step 2: calculate the three exponentials

LaTeX: e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283

Step 3: add them to get one common denominator

LaTeX: D=0.367879441+1+0.135335283=1.503214724

Step 4: divide each exponential by this denominator

LaTeX: p_1=\frac{0.367879441}{1.503214724}=0.244728471

LaTeX: p_2=\frac{1}{1.503214724}=0.665240956

LaTeX: p_3=\frac{0.135335283}{1.503214724}=0.090030573

The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability.

Step 5: use the probability of the correct class in CCE

LaTeX: L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2

LaTeX: L=-\ln(0.665240956)=0.407605964

Final answer: probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606.

Output delta for half squared loss

Delta is a short name for the loss derivative with respect to a neuron’s raw input z. For half squared loss, multiply prediction minus target by the activation’s derivative. The result is then multiplied by an incoming activation to obtain a weight gradient.

LaTeX: \delta^{out}=(o-y)f\prime(z)

Loss scaling: derive the gradient instead of guessing a factor

1. One error, three possible objectives

Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square:

Diagram: Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.. Labels: Prediction error and its derivative | Prediction error and its derivative | Keep the target y fixed when differentiating. | Half squared loss | L = ½(ŷ − y)² | Derivative = ŷ − y | The 2 cancels the half. | Mean squared error | L = sum of squares ÷ n | Derivative per output | = (2/n)(ŷ − y) | The scale of the loss changes the scale of its gradient.

LaTeX: \frac{\partial (a-y)^2}{\partial a}=2(a-y)\times1

Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below.

2. Why only the matching output appears

Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor.

Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages.

3. The activation contributes another factor

If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore:

LaTeX: \delta=\frac{\partial L}{\partial z}=(a-y)a(1-a)

For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question.

4. A numerical comparison

For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042.

The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used.

Complete sigmoid training example

Question. A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update.

Given

x = 2; w = 0; b = 0; y = 1.

Sigmoid output; half squared loss; η = 0.1.

To find

Find the weight-gradient sign, exact weight update and resulting weight.

Formula — which one and why

Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.

LaTeX: z=\sum_iw_ix_i+b

LaTeX: a=f(z)

Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

LaTeX: p=\frac{1}{1+e^{-z}}

LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.

LaTeX: L=\frac12\sum_j(o_j-y_j)^2

LaTeX: \frac{\partial L}{\partial o_j}=o_j-y_j

Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Step 1 forward: net = 0(2) + 0 = 0; sigmoid output = 0.5.

Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input.

Step 2 backward: prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125.

Step 3 gradients and update: weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree.

Drill 10 Sigmoid neuron gradient

Answer — step by step

Given

x = 2; w = 0.5; b = 0; target y = 1.

Activation: sigmoid; loss: half squared error.

Learning rate η = 0.1.

To find

Find the output, delta, weight/bias gradients and one update.

Formula — which one and why

Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.

LaTeX: z=\sum_iw_ix_i+b

LaTeX: a=f(z)

Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.

Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).

z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.

LaTeX: p=\frac{1}{1+e^{-z}}

LaTeX: \frac{dp}{dz}=p(1-p)

Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).

Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.

oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.

LaTeX: L=\frac12\sum_j(o_j-y_j)^2

LaTeX: \frac{\partial L}{\partial o_j}=o_j-y_j

Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.

Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.

δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.

LaTeX: \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j})

LaTeX: \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}

LaTeX: \frac{\partial L}{\partial W_{ji}}=\delta_ja_i

LaTeX: \frac{\partial L}{\partial b_j}=\delta_j

LaTeX: W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}}

Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.

Solution — step by step

Meaning and formulas

Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly.

LaTeX: z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2

The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z.

LaTeX: \delta=(p-y)p(1-p)

LaTeX: \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta

LaTeX: w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta

Step 1: weighted sum and sigmoid

LaTeX: z=0.5\times2+0=1

LaTeX: p=\frac1{1+e^{-1}}=0.731058579

Step 2: error and activation slope

LaTeX: p-y=0.731058579-1=-0.268941421

LaTeX: p(1-p)=0.731058579\times0.268941421=0.196611933

Step 3: multiply the linked effects to obtain delta

LaTeX: \delta=(-0.268941421)\times0.196611933=-0.052877093

Step 4: multiply delta by the input to each parameter

The weight receives x = 2. The bias effectively receives the constant input 1.

LaTeX: \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186

LaTeX: \frac{\partial L}{\partial b}=-0.052877093

Step 5: subtract learning rate times each gradient

LaTeX: w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419

LaTeX: b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709

Final answer: new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here.